2024-03-29T11:07:50.866 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:07:50.866 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --verbose --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693 --owner scheduled_amathuri@teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693/orig.config.yaml 2024-03-29T11:07:50.901 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:07:50.985 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622693' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOWEyAUDB6nKarlTJ77qio03BlccevtgBeAzfG88npriw1GXRuGLuaygY7KdT1Cok+NjmfiQsE7OoEF4dEyVwKc= smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPnLy+p2JFVioUa8/WLiHzMs21gwvt5OFzRAMN/H+2A0+xE4eK/hll2PJ4DvI6L8aWJb9ENYFvMXZgSywlqtsw0= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:07:50.986 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:07:50.986 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:07:50.987 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:07:50.988 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:07:51.012 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:07:51.012 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:07:51.012 INFO:teuthology.packaging:ref: None 2024-03-29T11:07:51.012 INFO:teuthology.packaging:tag: None 2024-03-29T11:07:51.012 INFO:teuthology.packaging:branch: main 2024-03-29T11:07:51.012 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:07:51.012 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:07:51.369 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:07:51.371 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:07:51.383 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:07:51.383 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:07:51.398 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:07:51.414 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:07:51.425 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:07:51.448 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:08.879224', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOWEyAUDB6nKarlTJ77qio03BlccevtgBeAzfG88npriw1GXRuGLuaygY7KdT1Cok+NjmfiQsE7OoEF4dEyVwKc='} 2024-03-29T11:07:51.467 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:08.877644', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPnLy+p2JFVioUa8/WLiHzMs21gwvt5OFzRAMN/H+2A0+xE4eK/hll2PJ4DvI6L8aWJb9ENYFvMXZgSywlqtsw0='} 2024-03-29T11:07:51.467 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:07:51.479 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:07:51.479 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:07:51.479 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:07:51.556 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbe9ab060d0>, signals=[15]) 2024-03-29T11:07:51.557 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:07:51.570 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:07:51.570 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-03-29T11:07:51.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:07:51.678 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T11:07:51.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:07:51.770 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:07:51.780 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-03-29T11:07:51.799 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-03-29T11:07:51.799 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-03-29T11:07:51.856 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-03-29T11:07:51.857 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-03-29T11:07:51.857 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-03-29T11:07:51.857 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:07:51.857 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-03-29T11:07:51.857 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:07:51.857 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:07:51.857 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-03-29T11:07:51.857 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:07:51.857 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:07:51.857 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:07:51.857 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:07:51.858 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:07:51.858 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:07:51.858 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-03-29T11:07:51.879 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2024-03-29T11:07:51.897 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2024-03-29T11:07:51.897 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2024-03-29T11:07:51.954 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="CentOS Stream" 2024-03-29T11:07:51.954 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="9" 2024-03-29T11:07:51.954 INFO:teuthology.orchestra.run.smithi082.stdout:ID="centos" 2024-03-29T11:07:51.954 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:07:51.954 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="9" 2024-03-29T11:07:51.954 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:07:51.954 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:07:51.954 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31" 2024-03-29T11:07:51.954 INFO:teuthology.orchestra.run.smithi082.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:07:51.954 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:07:51.954 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:07:51.954 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:07:51.955 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:07:51.955 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:07:51.955 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2024-03-29T11:07:51.979 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:07:51.992 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:07:52.017 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:07:52.018 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:07:52.020 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:07:52.036 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:07:52.042 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:07:52.042 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:07:52.077 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:07:52.104 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:07:52.153 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:07:52.169 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:07:52.169 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:07:52.169 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:07:52.169 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:07:52.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:52.186 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-03-29T11:07:52.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:52.187 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-03-29T11:07:52.243 INFO:teuthology.orchestra.run.smithi082.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:07:52.243 INFO:teuthology.task.kernel:Running kernel on smithi082: 5.14.0-361.el9.x86_64 2024-03-29T11:07:52.244 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum install -y kernel 2024-03-29T11:07:52.247 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:07:52.247 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.14.0-361.el9.x86_64 2024-03-29T11:07:52.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2024-03-29T11:07:54.372 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:07:54.400 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T11:07:55.806 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-03-29T11:07:55.808 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-29T11:07:56.799 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-29T11:07:57.813 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.0 MB 00:01 2024-03-29T11:08:00.273 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-29T11:08:11.823 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 798 kB/s | 6.1 MB 00:07 2024-03-29T11:08:13.432 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-29T11:08:13.972 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-03-29T11:08:14.927 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-29T11:08:18.547 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.1 MB 00:00 2024-03-29T11:08:19.964 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 16 kB 00:00 2024-03-29T11:08:20.083 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-29T11:08:22.441 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:08:22.441 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:08:22.530 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:08:22.539 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:08:22.539 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-29T11:08:22.539 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:08:22.540 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:08:22.540 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:08:22.540 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-29T11:08:22.540 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:08:22.540 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:08:22.540 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:08:22.540 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:08:22.540 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:08:22.540 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:08:22.540 INFO:teuthology.orchestra.run.smithi067.stdout:Install 4 Packages 2024-03-29T11:08:22.540 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:08:22.541 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 76 M 2024-03-29T11:08:22.541 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 123 M 2024-03-29T11:08:22.541 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:08:22.768 INFO:teuthology.orchestra.run.smithi067.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 846 kB/s | 192 kB 00:00 2024-03-29T11:08:22.957 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 7.2 MB/s | 21 MB 00:02 2024-03-29T11:08:23.576 INFO:teuthology.orchestra.run.smithi067.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 15 MB/s | 15 MB 00:01 2024-03-29T11:08:24.260 INFO:teuthology.orchestra.run.smithi067.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 20 MB/s | 34 MB 00:01 2024-03-29T11:08:24.735 INFO:teuthology.orchestra.run.smithi067.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 14 MB/s | 28 MB 00:01 2024-03-29T11:08:24.738 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:24.738 INFO:teuthology.orchestra.run.smithi067.stdout:Total 35 MB/s | 76 MB 00:02 2024-03-29T11:08:24.739 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:08:25.159 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:08:25.159 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:08:26.026 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:08:26.026 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:08:28.057 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T11:08:28.447 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:08:29.092 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:29.194 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:30.380 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:08:30.380 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:08:30.466 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:08:30.474 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:08:30.475 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T11:08:30.475 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:08:30.475 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:08:30.475 INFO:teuthology.orchestra.run.smithi082.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:08:30.475 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T11:08:30.475 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:08:30.475 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:08:30.475 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:08:30.475 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:08:30.475 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:08:30.475 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:08:30.475 INFO:teuthology.orchestra.run.smithi082.stdout:Install 4 Packages 2024-03-29T11:08:30.475 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:08:30.476 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 76 M 2024-03-29T11:08:30.476 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 123 M 2024-03-29T11:08:30.476 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:08:30.633 INFO:teuthology.orchestra.run.smithi082.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:08:31.358 INFO:teuthology.orchestra.run.smithi082.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-03-29T11:08:31.875 INFO:teuthology.orchestra.run.smithi082.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-03-29T11:08:32.086 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:32.175 INFO:teuthology.orchestra.run.smithi082.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 18 MB/s | 28 MB 00:01 2024-03-29T11:08:32.178 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:32.178 INFO:teuthology.orchestra.run.smithi082.stdout:Total 45 MB/s | 76 MB 00:01 2024-03-29T11:08:32.178 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:08:32.220 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:32.542 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:08:32.542 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:08:33.439 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:08:33.440 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:08:35.742 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:08:36.562 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:36.641 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:37.323 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:37.361 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:39.366 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:39.512 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:42.483 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:44.214 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:44.293 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:49.148 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:13.514 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:13.515 INFO:teuthology.orchestra.run.smithi067.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:09:13.515 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:09:13.559 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:13.977 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:13.977 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:13.977 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:13.977 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:14.402 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:14.402 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:09:14.403 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:09:14.403 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:14.403 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:14.403 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:09:14.403 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:14.403 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:09:14.403 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:09:14.727 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:14.831 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:14.831 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:14.831 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:09:14.831 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:09:14.832 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:09:14.832 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:09:14.832 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:09:14.832 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:14.915 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:14.915 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:14.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2024-03-29T11:09:14.962 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2024-03-29T11:09:14.962 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2024-03-29T11:09:14.962 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.06 2024-03-29T11:09:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 68.el9 2024-03-29T11:09:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2024-03-29T11:09:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:09:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:Group : Unspecified 2024-03-29T11:09:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 8264254 2024-03-29T11:09:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2024-03-29T11:09:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:09:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:09:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:09:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:09:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : builder@centos.org 2024-03-29T11:09:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2024-03-29T11:09:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:09:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2024-03-29T11:09:14.964 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2024-03-29T11:09:14.964 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:09:14.964 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:09:14.964 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:09:14.964 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:09:14.964 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2024-03-29T11:09:14.964 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:09:14.964 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:09:14.965 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:09:14.965 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:09:14.965 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:09:15.525 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-03-29T11:09:17.036 INFO:teuthology.orchestra.run.smithi067.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:09:17.071 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-03-29T11:09:17.073 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-03-29T11:09:17.091 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.Z1p2mneTPQ 2024-03-29T11:09:17.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Z1p2mneTPQ 2024-03-29T11:09:17.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.Z1p2mneTPQ 2024-03-29T11:09:17.329 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.Z1p2mneTPQ is 6KB 2024-03-29T11:09:17.354 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.Z1p2mneTPQ 2024-03-29T11:09:17.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:09:17.437 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:09:17.438 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:09:17.438 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:09:17.438 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:09:17.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:09:17.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-03-29T11:09:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:18.292 INFO:teuthology.orchestra.run.smithi082.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:09:18.292 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:09:18.335 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:19.167 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:19.167 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:09:19.168 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:09:19.168 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:19.168 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:19.168 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:09:19.168 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:19.168 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:09:19.168 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:09:19.516 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:19.618 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:19.618 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:19.618 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:09:19.618 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:09:19.618 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:09:19.619 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:09:19.619 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi082.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:09:19.619 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:19.698 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:19.699 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:19.699 DEBUG:teuthology.orchestra.run.smithi082:> sudo rpm -qi grub2-tools 2024-03-29T11:09:19.733 INFO:teuthology.orchestra.run.smithi082.stdout:Name : grub2-tools 2024-03-29T11:09:19.733 INFO:teuthology.orchestra.run.smithi082.stdout:Epoch : 1 2024-03-29T11:09:19.733 INFO:teuthology.orchestra.run.smithi082.stdout:Version : 2.06 2024-03-29T11:09:19.733 INFO:teuthology.orchestra.run.smithi082.stdout:Release : 68.el9 2024-03-29T11:09:19.733 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: x86_64 2024-03-29T11:09:19.733 INFO:teuthology.orchestra.run.smithi082.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:09:19.733 INFO:teuthology.orchestra.run.smithi082.stdout:Group : Unspecified 2024-03-29T11:09:19.733 INFO:teuthology.orchestra.run.smithi082.stdout:Size : 8264254 2024-03-29T11:09:19.733 INFO:teuthology.orchestra.run.smithi082.stdout:License : GPLv3+ 2024-03-29T11:09:19.733 INFO:teuthology.orchestra.run.smithi082.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:09:19.734 INFO:teuthology.orchestra.run.smithi082.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:09:19.734 INFO:teuthology.orchestra.run.smithi082.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:09:19.734 INFO:teuthology.orchestra.run.smithi082.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:09:19.734 INFO:teuthology.orchestra.run.smithi082.stdout:Packager : builder@centos.org 2024-03-29T11:09:19.734 INFO:teuthology.orchestra.run.smithi082.stdout:Vendor : CentOS 2024-03-29T11:09:19.734 INFO:teuthology.orchestra.run.smithi082.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:09:19.734 INFO:teuthology.orchestra.run.smithi082.stdout:Summary : Support tools for GRUB. 2024-03-29T11:09:19.734 INFO:teuthology.orchestra.run.smithi082.stdout:Description : 2024-03-29T11:09:19.734 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:09:19.734 INFO:teuthology.orchestra.run.smithi082.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:09:19.734 INFO:teuthology.orchestra.run.smithi082.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:09:19.734 INFO:teuthology.orchestra.run.smithi082.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:09:19.734 INFO:teuthology.orchestra.run.smithi082.stdout:hardware devices. 2024-03-29T11:09:19.734 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:09:19.735 INFO:teuthology.orchestra.run.smithi082.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:09:19.735 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:09:19.735 INFO:teuthology.task.kernel:Updating grub on smithi082 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:09:19.736 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:09:20.367 INFO:teuthology.orchestra.run.smithi082.stderr:Generating grub configuration file ... 2024-03-29T11:09:21.749 INFO:teuthology.orchestra.run.smithi082.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:09:21.782 INFO:teuthology.orchestra.run.smithi082.stderr:done 2024-03-29T11:09:21.784 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-03-29T11:09:21.798 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.8sKDk3dViu 2024-03-29T11:09:21.799 DEBUG:teuthology.orchestra.run.smithi082:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8sKDk3dViu 2024-03-29T11:09:21.867 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.8sKDk3dViu 2024-03-29T11:09:22.006 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.8sKDk3dViu is 6KB 2024-03-29T11:09:22.055 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.8sKDk3dViu 2024-03-29T11:09:22.069 DEBUG:teuthology.orchestra.run.smithi082:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:09:22.135 INFO:teuthology.orchestra.run.smithi082.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:09:22.135 INFO:teuthology.orchestra.run.smithi082.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:09:22.135 INFO:teuthology.orchestra.run.smithi082.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:09:22.136 INFO:teuthology.orchestra.run.smithi082.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:09:22.137 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:09:22.297 DEBUG:teuthology.orchestra.run.smithi082:> sudo shutdown -r now 2024-03-29T11:09:47.600 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:09:47.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-29T11:09:47.601 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:52.302 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:09:52.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-29T11:09:52.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:06.110 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-03-29T11:10:10.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-03-29T11:10:15.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-29T11:10:15.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:18.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-03-29T11:10:19.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-29T11:10:19.722 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:22.782 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-03-29T11:10:30.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-29T11:10:30.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:34.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-29T11:10:34.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:48.607 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-03-29T11:10:53.214 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-03-29T11:11:03.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-29T11:11:03.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:04.033 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-03-29T11:11:04.419 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-29T11:11:04.419 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:11:04.419 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-03-29T11:11:04.433 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:11:04.434 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:11:04.434 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:11:04.434 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:11:05.434 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:11:05.434 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:11:05.687 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-03-29T11:11:05.745 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:08.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-29T11:11:08.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:08.597 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-03-29T11:11:09.083 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-29T11:11:09.084 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:11:09.084 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-03-29T11:11:09.098 INFO:teuthology.orchestra.run.smithi082.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:11:09.099 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:11:09.099 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:11:09.099 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:11:10.100 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:11:10.100 DEBUG:teuthology.orchestra.run.smithi082:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:11:10.203 INFO:teuthology.orchestra.run.smithi082.stdout:ttyS1 2024-03-29T11:11:10.234 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:10.237 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:11:10.246 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:11:10.247 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:11:10.249 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:11:10.276 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:11:10.284 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:11:10.296 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:11:10.296 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:11:10.311 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:11:10.351 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:11:10.360 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:11:10.360 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:11:10.375 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:11:10.410 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:10.423 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:10.441 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:10.460 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:10.462 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:11:10.472 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:11:10.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:11:10.474 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:11:10.518 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:11:10.529 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:11:10.530 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:11:10.548 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:11:10.585 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:10.647 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:10.703 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:10.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:10.835 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:11:10.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:11:10.905 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:10.943 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:11.003 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:11.080 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:11.127 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:11:11.128 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:11:11.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-03-29T11:11:11.197 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2024-03-29T11:11:11.251 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:11:11.345 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:11:11.799 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:11:11.806 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:11:11.807 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:11:11.827 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:11:11.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-03-29T11:11:11.919 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-03-29T11:11:11.921 DEBUG:teuthology.orchestra.run.smithi082:> sudo service auditd rotate 2024-03-29T11:11:11.973 INFO:teuthology.orchestra.run.smithi082.stdout:Rotating logs: 2024-03-29T11:11:11.974 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:11:11.974 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-03-29T11:11:11.998 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-03-29T11:11:11.998 DEBUG:teuthology.orchestra.run.smithi082:> /usr/sbin/getenforce 2024-03-29T11:11:12.021 INFO:teuthology.orchestra.run.smithi082.stdout:Permissive 2024-03-29T11:11:12.021 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi082.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:11:12.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:11:12.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:11:12.070 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:11:12.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:11:12.097 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:11:12.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:11:12.158 DEBUG:teuthology.orchestra.run.smithi082:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:11:12.190 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:11:12.200 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:11:12.218 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:11:12.219 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi082.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:15:39.879 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi082.front.sepia.ceph.com')] 2024-03-29T11:15:39.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-29T11:15:39.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:15:39.980 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-03-29T11:15:40.057 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-29T11:15:40.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-29T11:15:40.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:15:40.146 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-03-29T11:15:40.226 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-29T11:15:40.226 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:15:40.236 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:15:40.237 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:15:40.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:15:40.239 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:15:40.240 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:15:40.269 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:15:40.286 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:15:40.298 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:15:40.315 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:15:40.316 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-03-29T11:15:40.330 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-03-29T11:15:40.343 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: ntpd: command not found 2024-03-29T11:15:40.347 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:15:40.357 INFO:teuthology.orchestra.run.smithi082.stdout:506 Cannot talk to daemon 2024-03-29T11:15:40.364 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:15:40.373 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:15:40.389 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:15:40.415 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:15:40.419 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:15:40.419 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-03-29T11:15:40.419 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:40.419 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:40.419 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:40.419 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:40.445 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:15:40.448 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:15:40.448 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-03-29T11:15:40.448 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:40.448 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:40.448 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:40.448 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:40.449 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:15:40.457 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:15:40.458 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:15:40.458 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:15:40.458 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:15:40.464 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:15:40.464 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:15:40.464 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:15:40.465 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:15:40.467 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:15:40.468 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:15:40.611 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:15:40.611 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:15:40.619 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:15:40.619 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:15:40.722 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:15:40.722 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:15:40.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:15:40.751 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:15:40.751 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:15:40.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:15:40.841 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:15:40.841 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:15:40.841 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:15:40.870 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:15:40.870 DEBUG:teuthology.orchestra.run.smithi082:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:15:40.899 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:15:40.939 DEBUG:teuthology.orchestra.run.smithi082:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:15:40.965 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-03-29T11:15:40.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-03-29T11:15:41.019 DEBUG:teuthology.orchestra.run.smithi082:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:15:41.085 INFO:teuthology.orchestra.run.smithi082.stdout:check_obsoletes = 1 2024-03-29T11:15:41.087 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum clean all 2024-03-29T11:15:41.203 INFO:teuthology.orchestra.run.smithi067.stdout:74 files removed 2024-03-29T11:15:41.230 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw 2024-03-29T11:15:41.345 INFO:teuthology.orchestra.run.smithi082.stdout:74 files removed 2024-03-29T11:15:41.372 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-radosgw 2024-03-29T11:15:41.851 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 260 kB/s | 82 kB 00:00 2024-03-29T11:15:42.082 INFO:teuthology.orchestra.run.smithi082.stdout:ceph packages for x86_64 200 kB/s | 82 kB 00:00 2024-03-29T11:15:42.310 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-29T11:15:42.449 INFO:teuthology.orchestra.run.smithi082.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-29T11:15:42.744 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-03-29T11:15:42.791 INFO:teuthology.orchestra.run.smithi082.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-03-29T11:15:43.260 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T11:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - AppStream 7.3 MB/s | 6.7 MB 00:00 2024-03-29T11:15:44.652 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-29T11:15:45.191 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-29T11:15:45.384 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.0 MB 00:00 2024-03-29T11:15:46.323 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-29T11:15:48.783 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-29T11:15:50.469 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - AppStream 4.8 MB/s | 19 MB 00:03 2024-03-29T11:15:55.600 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.1 MB 00:03 2024-03-29T11:15:57.001 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 16 kB 00:00 2024-03-29T11:15:57.702 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.1 MB 00:03 2024-03-29T11:15:58.671 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-29T11:15:59.494 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-29T11:16:00.880 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-29T11:16:03.853 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T11:16:05.828 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T11:16:06.552 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:16:06.555 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:16:06.555 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-29T11:16:06.556 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:16:06.556 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:16:06.556 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:16:06.556 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-03-29T11:16:06.556 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:16:06.556 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:16:06.556 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-29T11:16:06.556 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options 2024-03-29T11:16:06.556 INFO:teuthology.orchestra.run.smithi067.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:16:06.556 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:16:06.556 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:16:06.556 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:16:06.556 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:16:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:16:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:16:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:16:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:16:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:16:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:16:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:16:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:16:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:16:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:16:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:16:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:16:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse 2024-03-29T11:16:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:16:06.558 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:16:06.558 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:16:06.558 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:16:06.558 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:16:06.558 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:16:06.558 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:16:06.558 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:16:06.558 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:16:06.558 INFO:teuthology.orchestra.run.smithi067.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:16:06.558 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:16:06.558 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-03-29T11:16:06.558 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:16:06.558 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:16:06.558 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:16:06.559 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:16:06.559 INFO:teuthology.orchestra.run.smithi067.stdout:Install 30 Packages 2024-03-29T11:16:06.559 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 2 Packages 2024-03-29T11:16:06.559 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:16:06.559 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 58 M 2024-03-29T11:16:06.559 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:16:08.429 INFO:teuthology.orchestra.run.smithi067.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 5.4 MB/s | 9.7 MB 00:01 2024-03-29T11:16:08.445 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:16:08.470 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:16:08.470 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T11:16:08.470 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:16:08.470 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:16:08.470 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:16:08.470 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrading: 2024-03-29T11:16:08.470 INFO:teuthology.orchestra.run.smithi082.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:16:08.471 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:16:08.472 INFO:teuthology.orchestra.run.smithi082.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:16:08.473 INFO:teuthology.orchestra.run.smithi082.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:16:08.473 INFO:teuthology.orchestra.run.smithi082.stdout:Installing weak dependencies: 2024-03-29T11:16:08.473 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:16:08.473 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:16:08.473 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:16:08.473 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:16:08.473 INFO:teuthology.orchestra.run.smithi082.stdout:Install 30 Packages 2024-03-29T11:16:08.473 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrade 2 Packages 2024-03-29T11:16:08.473 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:16:08.473 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 58 M 2024-03-29T11:16:08.474 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:16:08.479 INFO:teuthology.orchestra.run.smithi067.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 505 kB/s | 25 kB 00:00 2024-03-29T11:16:08.604 INFO:teuthology.orchestra.run.smithi067.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.8 MB/s | 742 kB 00:00 2024-03-29T11:16:08.687 INFO:teuthology.orchestra.run.smithi067.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 5.9 MB/s | 498 kB 00:00 2024-03-29T11:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout:(5/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 8.0 MB/s | 20 MB 00:02 2024-03-29T11:16:09.197 INFO:teuthology.orchestra.run.smithi067.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 493 kB/s | 45 kB 00:00 2024-03-29T11:16:09.305 INFO:teuthology.orchestra.run.smithi067.stdout:(7/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 8.3 MB/s | 5.1 MB 00:00 2024-03-29T11:16:09.331 INFO:teuthology.orchestra.run.smithi067.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 983 kB/s | 131 kB 00:00 2024-03-29T11:16:09.398 INFO:teuthology.orchestra.run.smithi067.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.7 MB/s | 162 kB 00:00 2024-03-29T11:16:09.431 INFO:teuthology.orchestra.run.smithi067.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 3.1 MB/s | 320 kB 00:00 2024-03-29T11:16:09.531 INFO:teuthology.orchestra.run.smithi067.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 992 kB/s | 99 kB 00:00 2024-03-29T11:16:09.640 INFO:teuthology.orchestra.run.smithi067.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2024-03-29T11:16:09.682 INFO:teuthology.orchestra.run.smithi067.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-29T11:16:09.707 INFO:teuthology.orchestra.run.smithi067.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-29T11:16:09.757 INFO:teuthology.orchestra.run.smithi067.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-29T11:16:09.799 INFO:teuthology.orchestra.run.smithi067.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-29T11:16:09.812 INFO:teuthology.orchestra.run.smithi082.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 4.7 MB/s | 5.3 MB 00:01 2024-03-29T11:16:09.825 INFO:teuthology.orchestra.run.smithi067.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-29T11:16:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 6.5 MB/s | 220 kB 00:00 2024-03-29T11:16:09.884 INFO:teuthology.orchestra.run.smithi067.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.4 MB/s | 35 kB 00:00 2024-03-29T11:16:09.895 INFO:teuthology.orchestra.run.smithi082.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 301 kB/s | 25 kB 00:00 2024-03-29T11:16:09.934 INFO:teuthology.orchestra.run.smithi067.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 927 kB/s | 46 kB 00:00 2024-03-29T11:16:09.976 INFO:teuthology.orchestra.run.smithi067.stdout:(21/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 515 kB/s | 298 kB 00:00 2024-03-29T11:16:10.012 INFO:teuthology.orchestra.run.smithi082.stdout:(3/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 7.2 MB/s | 9.7 MB 00:01 2024-03-29T11:16:10.046 INFO:teuthology.orchestra.run.smithi082.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 4.8 MB/s | 742 kB 00:00 2024-03-29T11:16:10.079 INFO:teuthology.orchestra.run.smithi082.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 7.3 MB/s | 498 kB 00:00 2024-03-29T11:16:10.155 INFO:teuthology.orchestra.run.smithi082.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 601 kB/s | 45 kB 00:00 2024-03-29T11:16:10.246 INFO:teuthology.orchestra.run.smithi082.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.4 MB/s | 131 kB 00:00 2024-03-29T11:16:10.347 INFO:teuthology.orchestra.run.smithi082.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.6 MB/s | 162 kB 00:00 2024-03-29T11:16:10.351 INFO:teuthology.orchestra.run.smithi067.stdout:(22/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x8 1.4 MB/s | 5.3 MB 00:03 2024-03-29T11:16:10.376 INFO:teuthology.orchestra.run.smithi067.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 698 kB/s | 309 kB 00:00 2024-03-29T11:16:10.443 INFO:teuthology.orchestra.run.smithi067.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 727 kB/s | 49 kB 00:00 2024-03-29T11:16:10.447 INFO:teuthology.orchestra.run.smithi082.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.1 MB/s | 320 kB 00:00 2024-03-29T11:16:10.527 INFO:teuthology.orchestra.run.smithi067.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 809 kB/s | 67 kB 00:00 2024-03-29T11:16:10.547 INFO:teuthology.orchestra.run.smithi082.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.9 MB/s | 298 kB 00:00 2024-03-29T11:16:10.602 INFO:teuthology.orchestra.run.smithi067.stdout:(26/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 103 kB/s | 26 kB 00:00 2024-03-29T11:16:10.623 INFO:teuthology.orchestra.run.smithi082.stdout:(11/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 8.8 MB/s | 5.1 MB 00:00 2024-03-29T11:16:10.648 INFO:teuthology.orchestra.run.smithi082.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 990 kB/s | 99 kB 00:00 2024-03-29T11:16:10.686 INFO:teuthology.orchestra.run.smithi067.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.2 MB/s | 4.4 MB 00:00 2024-03-29T11:16:10.719 INFO:teuthology.orchestra.run.smithi067.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-03-29T11:16:10.773 INFO:teuthology.orchestra.run.smithi082.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 705 kB/s | 106 kB 00:00 2024-03-29T11:16:10.786 INFO:teuthology.orchestra.run.smithi067.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-29T11:16:10.798 INFO:teuthology.orchestra.run.smithi082.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-29T11:16:10.815 INFO:teuthology.orchestra.run.smithi082.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T11:16:10.828 INFO:teuthology.orchestra.run.smithi067.stdout:(30/32): re2-20211101-3.el9.x86_64.rpm 849 kB/s | 192 kB 00:00 2024-03-29T11:16:10.891 INFO:teuthology.orchestra.run.smithi082.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-29T11:16:10.916 INFO:teuthology.orchestra.run.smithi082.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-03-29T11:16:10.932 INFO:teuthology.orchestra.run.smithi082.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 995 kB/s | 41 kB 00:00 2024-03-29T11:16:10.958 INFO:teuthology.orchestra.run.smithi082.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T11:16:10.983 INFO:teuthology.orchestra.run.smithi082.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 695 kB/s | 35 kB 00:00 2024-03-29T11:16:11.009 INFO:teuthology.orchestra.run.smithi082.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 924 kB/s | 46 kB 00:00 2024-03-29T11:16:11.162 INFO:teuthology.orchestra.run.smithi067.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 8.3 MB/s | 3.1 MB 00:00 2024-03-29T11:16:11.201 INFO:teuthology.orchestra.run.smithi082.stdout:(22/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 7.8 MB/s | 20 MB 00:02 2024-03-29T11:16:11.251 INFO:teuthology.orchestra.run.smithi082.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 523 kB/s | 26 kB 00:00 2024-03-29T11:16:11.293 INFO:teuthology.orchestra.run.smithi082.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-03-29T11:16:11.318 INFO:teuthology.orchestra.run.smithi082.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 727 kB/s | 49 kB 00:00 2024-03-29T11:16:11.343 INFO:teuthology.orchestra.run.smithi082.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-29T11:16:11.418 INFO:teuthology.orchestra.run.smithi082.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-29T11:16:11.444 INFO:teuthology.orchestra.run.smithi082.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-29T11:16:11.485 INFO:teuthology.orchestra.run.smithi082.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-03-29T11:16:11.536 INFO:teuthology.orchestra.run.smithi082.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-29T11:16:12.211 INFO:teuthology.orchestra.run.smithi082.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 4.3 MB/s | 3.1 MB 00:00 2024-03-29T11:16:12.237 INFO:teuthology.orchestra.run.smithi067.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 2.2 MB/s | 3.4 MB 00:01 2024-03-29T11:16:12.244 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:12.244 INFO:teuthology.orchestra.run.smithi067.stdout:Total 10 MB/s | 58 MB 00:05 2024-03-29T11:16:12.244 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:16:12.286 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:16:12.287 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:16:12.303 INFO:teuthology.orchestra.run.smithi082.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 3.9 MB/s | 3.4 MB 00:00 2024-03-29T11:16:12.308 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:12.308 INFO:teuthology.orchestra.run.smithi082.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-29T11:16:12.308 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:16:12.345 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:16:12.345 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:16:12.769 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:16:12.769 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:16:12.821 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:16:12.821 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:16:14.258 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:16:14.285 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:16:14.338 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:16:14.372 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:16:14.375 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:16:14.401 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:16:14.415 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:16:14.423 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:14.440 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:14.441 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:16:14.464 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:14.481 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:14.631 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:14.685 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:14.704 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:15.009 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:16:15.045 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:16:15.059 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:15.102 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:15.150 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:15.177 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:15.237 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:16:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:16:15.269 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:15.331 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:15.358 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:15.360 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:16:15.387 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:16:15.403 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:16:15.431 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:16:15.471 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:15.496 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:15.559 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:15.586 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:15.594 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:16:15.621 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:16:15.701 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:16:15.731 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:16:15.737 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:16:15.772 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:16:15.777 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:16:15.810 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:16:15.813 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:16:15.839 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:16:15.845 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:16:15.872 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:16:15.875 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:16:15.907 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:16:16.177 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:16:16.210 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:16.236 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:16:16.270 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:16.299 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:16.332 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:16:16.360 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:16.371 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:16:16.392 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:16:16.431 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:16:16.823 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:16:16.846 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:16.907 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:16:16.917 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:16.931 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:16.985 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:16:16.985 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:17.015 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:16:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:16:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:16:19.185 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:19.228 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:19.334 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:19.363 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:19.776 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:19.814 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:19.890 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:19.890 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:16:19.890 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:16:19.890 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:16:19.923 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:19.961 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:20.092 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:20.092 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:16:20.092 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:16:20.092 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:16:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:29.415 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:29.415 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /sys 2024-03-29T11:16:29.415 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /proc 2024-03-29T11:16:29.415 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /mnt 2024-03-29T11:16:29.415 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /var/tmp 2024-03-29T11:16:29.415 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /home 2024-03-29T11:16:29.415 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /root 2024-03-29T11:16:29.415 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /tmp 2024-03-29T11:16:29.416 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:16:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:30.106 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-03-29T11:16:30.106 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-03-29T11:16:30.106 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-03-29T11:16:30.106 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-03-29T11:16:30.106 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-03-29T11:16:30.106 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-03-29T11:16:30.106 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-03-29T11:16:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:16:30.332 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:30.357 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:30.357 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:30.357 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:16:30.357 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:30.357 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:30.357 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:16:30.359 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:30.443 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:30.443 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:31.013 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:31.037 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:31.037 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:31.037 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:16:31.037 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:31.037 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:31.037 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:16:31.039 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:31.513 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:31.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:16:31.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:16:31.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:16:31.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:16:31.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:16:31.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:16:31.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:31.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:16:31.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:16:31.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:16:31.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:16:31.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:31.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:16:31.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:16:31.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:16:31.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:16:31.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:16:31.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:16:31.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:16:31.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:16:31.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:16:31.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:16:31.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:16:31.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:16:31.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:16:31.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:16:31.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:16:31.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:16:31.516 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:16:31.516 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:16:31.516 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:31.516 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:16:31.516 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:16:32.268 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:32.268 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:16:32.268 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:16:32.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:16:32.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:16:32.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:16:32.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:16:32.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:32.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:16:32.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:16:32.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:16:32.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:16:32.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:32.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:16:32.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:16:32.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:16:32.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:16:32.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:16:32.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:16:32.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:16:32.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:16:32.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:16:32.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:16:32.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:16:32.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:16:32.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:16:32.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:16:32.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:16:32.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:16:32.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:16:32.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:16:32.271 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:32.271 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:16:32.271 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:16:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:16:32.475 INFO:teuthology.orchestra.run.smithi082.stdout:Upgraded: 2024-03-29T11:16:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.475 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:16:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:16:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.475 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:16:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:16:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:16:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:16:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:16:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:16:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:16:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:16:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:16:32.476 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:16:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:16:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:16:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:16:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:16:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:16:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:16:32.477 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:16:32.477 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:16:32.812 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-test 2024-03-29T11:16:33.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:33.269 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:16:33.269 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-03-29T11:16:33.269 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.270 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.270 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:16:33.270 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:16:33.270 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.270 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.270 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.270 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.270 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:16:33.270 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:16:33.270 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:16:33.270 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.270 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:16:33.270 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:16:33.270 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:16:33.271 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:16:33.272 INFO:teuthology.orchestra.run.smithi067.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:16:33.272 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:16:33.272 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:16:33.272 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:16:33.368 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:16:33.519 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:16:33.520 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:16:33.520 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T11:16:33.520 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:16:33.520 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:16:33.520 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:16:33.520 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T11:16:33.521 INFO:teuthology.orchestra.run.smithi082.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:16:33.521 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:16:33.521 INFO:teuthology.orchestra.run.smithi082.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:16:33.521 INFO:teuthology.orchestra.run.smithi082.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:16:33.521 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:16:33.521 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:16:33.521 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:16:33.521 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:16:33.521 INFO:teuthology.orchestra.run.smithi082.stdout:Install 6 Packages 2024-03-29T11:16:33.521 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:16:33.522 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 48 M 2024-03-29T11:16:33.522 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 204 M 2024-03-29T11:16:33.522 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:16:33.597 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-test 2024-03-29T11:16:33.786 INFO:teuthology.orchestra.run.smithi082.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T11:16:33.828 INFO:teuthology.orchestra.run.smithi082.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-29T11:16:33.853 INFO:teuthology.orchestra.run.smithi082.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-29T11:16:33.996 INFO:teuthology.orchestra.run.smithi082.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-29T11:16:34.160 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:31 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:16:34.320 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:16:34.321 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:16:34.321 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-29T11:16:34.321 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:16:34.321 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:16:34.321 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:16:34.321 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-29T11:16:34.321 INFO:teuthology.orchestra.run.smithi067.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:16:34.321 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:16:34.321 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:16:34.321 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:16:34.322 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:16:34.322 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:16:34.322 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:16:34.322 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:16:34.322 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-03-29T11:16:34.322 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:16:34.322 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 48 M 2024-03-29T11:16:34.322 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 204 M 2024-03-29T11:16:34.323 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:16:34.354 INFO:teuthology.orchestra.run.smithi082.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 228 kB/s | 163 kB 00:00 2024-03-29T11:16:34.598 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:16:34.814 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 300 kB 00:00 2024-03-29T11:16:34.848 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-29T11:16:34.990 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-29T11:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 159 kB/s | 163 kB 00:01 2024-03-29T11:17:11.492 INFO:teuthology.orchestra.run.smithi082.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 47 MB 00:37 2024-03-29T11:17:11.495 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:11.495 INFO:teuthology.orchestra.run.smithi082.stdout:Total 1.3 MB/s | 48 MB 00:37 2024-03-29T11:17:11.508 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:17:11.528 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:17:11.528 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:17:11.914 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:17:11.914 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:17:12.485 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:17:12.559 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:17:12.602 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:17:12.642 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:12.683 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:17:12.711 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:15.787 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 47 MB 00:41 2024-03-29T11:17:15.790 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:15.790 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.2 MB/s | 48 MB 00:41 2024-03-29T11:17:15.803 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:17:15.826 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:17:15.826 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:17:16.220 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:17:16.220 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:17:16.809 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:17:16.889 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:17:16.931 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:17:16.971 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:17.013 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:17:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:17.159 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:17.236 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:19.594 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:19.594 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:17:19.594 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:17:19.594 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:19.594 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:17:19.594 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:17:19.940 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:17:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:19.941 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:17:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:17:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:17:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:17:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:17:19.941 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:19.941 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:17:20.230 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph 2024-03-29T11:17:20.796 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:15 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:17:20.956 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:17:20.958 INFO:teuthology.orchestra.run.smithi082.stdout:========================================================================================= 2024-03-29T11:17:20.958 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T11:17:20.958 INFO:teuthology.orchestra.run.smithi082.stdout:========================================================================================= 2024-03-29T11:17:20.958 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T11:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:17:20.959 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:17:20.960 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:17:20.961 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:17:20.962 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:17:20.962 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:17:20.962 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:20.962 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:17:20.962 INFO:teuthology.orchestra.run.smithi082.stdout:========================================================================================= 2024-03-29T11:17:20.962 INFO:teuthology.orchestra.run.smithi082.stdout:Install 36 Packages 2024-03-29T11:17:20.962 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:20.963 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 30 M 2024-03-29T11:17:20.963 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 107 M 2024-03-29T11:17:20.963 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:17:21.766 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:21.827 INFO:teuthology.orchestra.run.smithi082.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-03-29T11:17:21.843 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:22.286 INFO:teuthology.orchestra.run.smithi082.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 3.1 MB/s | 2.1 MB 00:00 2024-03-29T11:17:22.703 INFO:teuthology.orchestra.run.smithi082.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 1.5 MB 00:01 2024-03-29T11:17:22.894 INFO:teuthology.orchestra.run.smithi082.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.3 MB/s | 247 kB 00:00 2024-03-29T11:17:23.003 INFO:teuthology.orchestra.run.smithi082.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-29T11:17:23.053 INFO:teuthology.orchestra.run.smithi082.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-29T11:17:23.087 INFO:teuthology.orchestra.run.smithi082.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 945 kB/s | 32 kB 00:00 2024-03-29T11:17:23.121 INFO:teuthology.orchestra.run.smithi082.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:17:23.180 INFO:teuthology.orchestra.run.smithi082.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 661 kB/s | 39 kB 00:00 2024-03-29T11:17:23.221 INFO:teuthology.orchestra.run.smithi082.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:17:23.297 INFO:teuthology.orchestra.run.smithi082.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:17:23.372 INFO:teuthology.orchestra.run.smithi082.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-29T11:17:23.439 INFO:teuthology.orchestra.run.smithi082.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 1.8 MB/s | 124 kB 00:00 2024-03-29T11:17:23.606 INFO:teuthology.orchestra.run.smithi082.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.1 MB/s | 187 kB 00:00 2024-03-29T11:17:23.798 INFO:teuthology.orchestra.run.smithi082.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 651 kB/s | 125 kB 00:00 2024-03-29T11:17:23.998 INFO:teuthology.orchestra.run.smithi082.stdout:(16/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 9.8 MB/s | 17 MB 00:01 2024-03-29T11:17:24.023 INFO:teuthology.orchestra.run.smithi082.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 99 kB/s | 22 kB 00:00 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:24.187 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:17:24.187 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:17:24.257 INFO:teuthology.orchestra.run.smithi082.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 186 kB/s | 43 kB 00:00 2024-03-29T11:17:24.324 INFO:teuthology.orchestra.run.smithi082.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 464 kB/s | 151 kB 00:00 2024-03-29T11:17:24.516 INFO:teuthology.orchestra.run.smithi082.stdout:(20/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.7 MB/s | 4.7 MB 00:02 2024-03-29T11:17:24.541 INFO:teuthology.orchestra.run.smithi082.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 605 kB/s | 172 kB 00:00 2024-03-29T11:17:24.591 INFO:teuthology.orchestra.run.smithi082.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-03-29T11:17:24.608 INFO:teuthology.orchestra.run.smithi082.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 264 kB/s | 18 kB 00:00 2024-03-29T11:17:24.622 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:17:24.622 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:24.622 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:17:24.622 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:24.622 INFO:teuthology.orchestra.run.smithi067.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:17:24.622 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:24.622 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:17:24.622 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:17:24.622 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:17:24.622 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:24.622 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:17:24.659 INFO:teuthology.orchestra.run.smithi082.stdout:(24/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 75 kB/s | 11 kB 00:00 2024-03-29T11:17:24.675 INFO:teuthology.orchestra.run.smithi082.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 277 kB/s | 23 kB 00:00 2024-03-29T11:17:24.692 INFO:teuthology.orchestra.run.smithi082.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-03-29T11:17:24.718 INFO:teuthology.orchestra.run.smithi082.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 336 kB/s | 20 kB 00:00 2024-03-29T11:17:24.751 INFO:teuthology.orchestra.run.smithi082.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 794 kB/s | 46 kB 00:00 2024-03-29T11:17:24.776 INFO:teuthology.orchestra.run.smithi082.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 944 kB/s | 79 kB 00:00 2024-03-29T11:17:24.810 INFO:teuthology.orchestra.run.smithi082.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-03-29T11:17:24.844 INFO:teuthology.orchestra.run.smithi082.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-29T11:17:24.869 INFO:teuthology.orchestra.run.smithi082.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 612 kB/s | 36 kB 00:00 2024-03-29T11:17:24.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph 2024-03-29T11:17:24.927 INFO:teuthology.orchestra.run.smithi082.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-29T11:17:24.986 INFO:teuthology.orchestra.run.smithi082.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-29T11:17:25.003 INFO:teuthology.orchestra.run.smithi082.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-03-29T11:17:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.7 MB/s | 427 kB 00:00 2024-03-29T11:17:25.115 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:25.115 INFO:teuthology.orchestra.run.smithi082.stdout:Total 7.1 MB/s | 30 MB 00:04 2024-03-29T11:17:25.141 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:17:25.183 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:17:25.183 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:17:25.484 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:17:25.606 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:17:25.606 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:17:25.651 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:17:25.655 INFO:teuthology.orchestra.run.smithi067.stdout:========================================================================================= 2024-03-29T11:17:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-29T11:17:25.655 INFO:teuthology.orchestra.run.smithi067.stdout:========================================================================================= 2024-03-29T11:17:25.655 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:17:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:17:25.655 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-29T11:17:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:17:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:17:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:17:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:17:25.655 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:17:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:17:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:17:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:17:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:17:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:17:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:17:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:17:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:17:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:17:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:17:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:17:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:17:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:17:25.656 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:17:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:17:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:17:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:17:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:17:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:17:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:17:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:17:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:17:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:17:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:17:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:17:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:17:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:17:25.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:17:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:17:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:17:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:25.658 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:17:25.658 INFO:teuthology.orchestra.run.smithi067.stdout:========================================================================================= 2024-03-29T11:17:25.658 INFO:teuthology.orchestra.run.smithi067.stdout:Install 36 Packages 2024-03-29T11:17:25.658 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:25.659 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 30 M 2024-03-29T11:17:25.659 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 107 M 2024-03-29T11:17:25.659 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:17:26.574 INFO:teuthology.orchestra.run.smithi067.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 25 kB/s | 6.6 kB 00:00 2024-03-29T11:17:26.926 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:17:26.981 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:17:27.017 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:17:27.042 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:17:27.149 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:17:27.181 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:17:27.273 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:17:27.343 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:17:27.384 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:17:27.456 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:17:27.509 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:17:27.548 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:17:27.663 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:17:27.719 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:17:27.904 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:17:27.953 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:17:27.991 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:17:28.026 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:17:28.056 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:17:28.124 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:17:28.158 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:28.192 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:17:28.233 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:17:28.275 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:17:28.290 INFO:teuthology.orchestra.run.smithi067.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 769 kB/s | 1.5 MB 00:01 2024-03-29T11:17:28.318 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:17:28.407 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:17:28.441 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:17:28.512 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:17:28.641 INFO:teuthology.orchestra.run.smithi067.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 932 kB/s | 2.1 MB 00:02 2024-03-29T11:17:28.650 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:17:28.766 INFO:teuthology.orchestra.run.smithi067.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.9 MB/s | 247 kB 00:00 2024-03-29T11:17:28.775 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:28.893 INFO:teuthology.orchestra.run.smithi067.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T11:17:28.941 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:17:28.943 INFO:teuthology.orchestra.run.smithi067.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-29T11:17:28.971 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:28.977 INFO:teuthology.orchestra.run.smithi067.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 943 kB/s | 32 kB 00:00 2024-03-29T11:17:28.996 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:28.996 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:28.996 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:17:28.996 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:28.996 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:28.996 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:29.002 INFO:teuthology.orchestra.run.smithi067.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-29T11:17:29.028 INFO:teuthology.orchestra.run.smithi067.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-29T11:17:29.111 INFO:teuthology.orchestra.run.smithi067.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-03-29T11:17:29.295 INFO:teuthology.orchestra.run.smithi067.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 6.2 MB/s | 1.1 MB 00:00 2024-03-29T11:17:29.362 INFO:teuthology.orchestra.run.smithi067.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-29T11:17:29.404 INFO:teuthology.orchestra.run.smithi067.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-29T11:17:29.454 INFO:teuthology.orchestra.run.smithi067.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:17:29.680 INFO:teuthology.orchestra.run.smithi067.stdout:(15/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.5 MB/s | 4.7 MB 00:03 2024-03-29T11:17:29.788 INFO:teuthology.orchestra.run.smithi067.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-03-29T11:17:29.922 INFO:teuthology.orchestra.run.smithi067.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 267 kB/s | 125 kB 00:00 2024-03-29T11:17:29.964 INFO:teuthology.orchestra.run.smithi067.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 862 kB/s | 151 kB 00:00 2024-03-29T11:17:29.989 INFO:teuthology.orchestra.run.smithi067.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 653 kB/s | 43 kB 00:00 2024-03-29T11:17:30.014 INFO:teuthology.orchestra.run.smithi067.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-03-29T11:17:30.056 INFO:teuthology.orchestra.run.smithi067.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2024-03-29T11:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 161 kB/s | 11 kB 00:00 2024-03-29T11:17:30.106 INFO:teuthology.orchestra.run.smithi067.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-03-29T11:17:30.132 INFO:teuthology.orchestra.run.smithi067.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-03-29T11:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-03-29T11:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-03-29T11:17:30.208 INFO:teuthology.orchestra.run.smithi067.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 927 kB/s | 46 kB 00:00 2024-03-29T11:17:30.224 INFO:teuthology.orchestra.run.smithi067.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-03-29T11:17:30.258 INFO:teuthology.orchestra.run.smithi067.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-29T11:17:30.275 INFO:teuthology.orchestra.run.smithi067.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-03-29T11:17:30.300 INFO:teuthology.orchestra.run.smithi067.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-29T11:17:30.317 INFO:teuthology.orchestra.run.smithi067.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 854 kB/s | 36 kB 00:00 2024-03-29T11:17:30.342 INFO:teuthology.orchestra.run.smithi067.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-03-29T11:17:30.401 INFO:teuthology.orchestra.run.smithi067.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-29T11:17:30.426 INFO:teuthology.orchestra.run.smithi067.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-03-29T11:17:30.430 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:30.457 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:30.457 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:30.457 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:17:30.457 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:30.457 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:30.457 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:30.897 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:30.920 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:30.920 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:30.920 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:17:30.920 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:30.920 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:30.920 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:31.131 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:31.155 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:31.155 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:31.155 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:17:31.155 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:31.155 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:31.155 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:31.212 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:33.068 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:33.068 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:17:33.068 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:17:33.069 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:17:33.069 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:17:33.069 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:17:33.069 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:17:33.069 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:17:33.069 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:17:33.069 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:17:33.069 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:17:33.069 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:17:33.069 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:17:33.069 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:17:33.069 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:17:33.069 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:17:33.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:17:33.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:17:33.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:17:33.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:17:33.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:17:33.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:33.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:17:33.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:17:33.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:17:33.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:17:33.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:17:33.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:17:33.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:17:33.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:17:33.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:33.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:17:33.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:17:33.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:17:33.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:17:33.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:17:33.856 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:17:33.856 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:33.856 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:17:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:17:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:17:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:17:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:17:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:17:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:17:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:17:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:17:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:17:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:17:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:17:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:17:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:17:33.858 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:17:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:17:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:17:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:17:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:17:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:17:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:17:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:17:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:17:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:17:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:17:33.859 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:33.859 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:17:34.173 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-base 2024-03-29T11:17:34.736 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:17:34.834 INFO:teuthology.orchestra.run.smithi082.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:34.886 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:17:34.887 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T11:17:34.888 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:17:34.963 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install cephadm 2024-03-29T11:17:35.522 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:17:35.671 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:17:35.672 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:17:35.672 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T11:17:35.672 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:17:35.672 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:17:35.672 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:17:35.672 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:35.673 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:17:35.673 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:17:35.673 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-03-29T11:17:35.673 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:35.673 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 747 k 2024-03-29T11:17:35.673 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 754 k 2024-03-29T11:17:35.673 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:17:36.776 INFO:teuthology.orchestra.run.smithi082.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 678 kB/s | 747 kB 00:01 2024-03-29T11:17:36.776 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:36.776 INFO:teuthology.orchestra.run.smithi082.stdout:Total 677 kB/s | 747 kB 00:01 2024-03-29T11:17:36.777 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:17:36.783 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:17:36.783 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:17:36.789 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:17:36.790 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:17:36.869 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:17:37.305 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:37.390 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:37.784 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:38.050 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:38.051 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:38.051 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:17:38.051 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:38.051 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:38.051 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:17:38.202 INFO:teuthology.orchestra.run.smithi067.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.7 MB/s | 17 MB 00:09 2024-03-29T11:17:38.206 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:38.206 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.4 MB/s | 30 MB 00:12 2024-03-29T11:17:38.232 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:17:38.263 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:17:38.277 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:17:38.277 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:17:38.826 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:17:38.978 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:17:38.979 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:17:38.979 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-03-29T11:17:38.979 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:17:38.979 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:17:38.979 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:17:38.980 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:38.980 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:17:38.980 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:17:38.980 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-03-29T11:17:38.980 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:38.980 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 145 k 2024-03-29T11:17:38.980 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 436 k 2024-03-29T11:17:38.980 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:17:39.424 INFO:teuthology.orchestra.run.smithi082.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 328 kB/s | 145 kB 00:00 2024-03-29T11:17:39.424 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:39.425 INFO:teuthology.orchestra.run.smithi082.stdout:Total 326 kB/s | 145 kB 00:00 2024-03-29T11:17:39.425 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:17:39.432 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:17:39.433 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:17:39.485 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:17:39.485 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:17:39.622 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:17:39.691 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:39.715 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:39.715 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:39.715 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:17:39.715 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:40.059 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:17:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:17:40.163 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:17:40.190 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:17:40.265 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:17:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:17:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:17:40.444 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:17:40.528 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:17:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:17:40.692 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:40.692 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:40.692 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:17:40.692 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:40.692 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:40.692 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:17:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:17:40.747 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:17:40.806 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:17:40.869 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:17:40.943 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr 2024-03-29T11:17:41.067 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:17:41.118 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:17:41.156 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:17:41.190 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:17:41.220 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:17:41.301 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:17:41.330 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:41.363 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:17:41.405 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:17:41.449 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:17:41.482 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:17:41.514 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:17:41.578 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:17:41.612 INFO:teuthology.orchestra.run.smithi082.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:41.613 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:17:41.665 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:17:41.666 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T11:17:41.666 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:17:41.690 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:17:41.732 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:17:41.840 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:17:41.972 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:42.145 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:17:42.175 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:42.201 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:42.201 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:42.201 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:17:42.201 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:42.201 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:42.201 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:42.298 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:17:42.449 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:17:42.450 INFO:teuthology.orchestra.run.smithi082.stdout:====================================================================================== 2024-03-29T11:17:42.450 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T11:17:42.450 INFO:teuthology.orchestra.run.smithi082.stdout:====================================================================================== 2024-03-29T11:17:42.451 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:17:42.451 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:17:42.451 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T11:17:42.451 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:17:42.451 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:17:42.451 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:17:42.451 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:17:42.451 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:17:42.451 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:17:42.452 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:17:42.452 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:17:42.452 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:17:42.452 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:17:42.452 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:17:42.452 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:42.452 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:17:42.452 INFO:teuthology.orchestra.run.smithi082.stdout:====================================================================================== 2024-03-29T11:17:42.452 INFO:teuthology.orchestra.run.smithi082.stdout:Install 12 Packages 2024-03-29T11:17:42.452 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:42.453 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 6.9 M 2024-03-29T11:17:42.453 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 26 M 2024-03-29T11:17:42.453 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:17:43.336 INFO:teuthology.orchestra.run.smithi082.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 33 kB/s | 15 kB 00:00 2024-03-29T11:17:43.411 INFO:teuthology.orchestra.run.smithi082.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 59 kB/s | 31 kB 00:00 2024-03-29T11:17:43.586 INFO:teuthology.orchestra.run.smithi082.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-03-29T11:17:43.612 INFO:teuthology.orchestra.run.smithi082.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 238 kB 00:00 2024-03-29T11:17:43.726 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:43.752 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:43.752 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:17:43.753 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:43.753 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:43.845 INFO:teuthology.orchestra.run.smithi082.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.3 MB/s | 863 kB 00:00 2024-03-29T11:17:44.210 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:44.233 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:44.233 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:44.233 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:17:44.234 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:44.234 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:44.234 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:44.237 INFO:teuthology.orchestra.run.smithi082.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 50 kB/s | 19 kB 00:00 2024-03-29T11:17:44.404 INFO:teuthology.orchestra.run.smithi082.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 692 kB/s | 548 kB 00:00 2024-03-29T11:17:44.444 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:44.467 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:44.467 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:44.467 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:17:44.468 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:44.468 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:44.538 INFO:teuthology.orchestra.run.smithi082.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-29T11:17:44.646 INFO:teuthology.orchestra.run.smithi082.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-03-29T11:17:44.688 INFO:teuthology.orchestra.run.smithi082.stdout:(10/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.0 MB/s | 1.8 MB 00:01 2024-03-29T11:17:44.772 INFO:teuthology.orchestra.run.smithi082.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-29T11:17:44.922 INFO:teuthology.orchestra.run.smithi082.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.0 MB/s | 2.0 MB 00:00 2024-03-29T11:17:44.927 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:44.927 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.8 MB/s | 6.9 MB 00:02 2024-03-29T11:17:44.945 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:17:44.963 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:17:44.963 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:17:45.129 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:17:45.129 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:17:45.666 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:17:45.739 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:17:45.783 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:17:45.830 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:17:45.868 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:17:45.978 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:17:46.102 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:17:46.205 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:17:46.248 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:17:46.277 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:46.308 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:17:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:17:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:17:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:17:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:17:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:17:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:17:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:17:46.408 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:17:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:17:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:17:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:17:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:17:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:17:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:17:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:17:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:17:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:17:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:17:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:17:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:17:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:46.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:17:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:17:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:17:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:17:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:17:46.410 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:17:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:17:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:17:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:17:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:17:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:17:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:17:46.411 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:17:46.552 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:17:46.583 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:47.305 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:17:47.305 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:47.305 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:17:47.305 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:47.305 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:47.305 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:47.305 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:47.305 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:47.305 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:47.305 INFO:teuthology.orchestra.run.smithi067.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:17:47.305 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:17:47.305 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:17:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:17:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:17:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:17:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:17:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:17:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:17:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:17:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:17:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:17:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:17:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:17:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:17:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:17:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:17:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:17:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:17:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:17:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:17:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:17:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:17:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:17:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:17:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:17:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:17:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:17:47.308 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:17:47.308 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:17:47.308 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:47.308 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:17:47.375 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:47.375 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:17:47.375 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:17:47.375 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:17:47.376 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:17:47.376 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:17:47.376 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:17:47.376 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:17:47.376 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:17:47.376 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:47.376 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:17:47.376 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:17:47.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-base 2024-03-29T11:17:47.715 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:17:47.716 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:47.717 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:17:47.958 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:17:48.227 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:17:48.329 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:48.384 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:17:48.385 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-29T11:17:48.385 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:17:48.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install cephadm 2024-03-29T11:17:48.514 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:17:48.665 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:17:48.666 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================================ 2024-03-29T11:17:48.666 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T11:17:48.666 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================================ 2024-03-29T11:17:48.666 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:17:48.666 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:17:48.666 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T11:17:48.666 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:17:48.667 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:17:48.667 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:17:48.667 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:17:48.667 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:17:48.667 INFO:teuthology.orchestra.run.smithi082.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:17:48.667 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:17:48.667 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:17:48.667 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:17:48.667 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:17:48.667 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:17:48.667 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:48.667 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:17:48.667 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================================ 2024-03-29T11:17:48.668 INFO:teuthology.orchestra.run.smithi082.stdout:Install 12 Packages 2024-03-29T11:17:48.668 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:48.668 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 37 M 2024-03-29T11:17:48.668 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 209 M 2024-03-29T11:17:48.668 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:17:49.053 INFO:teuthology.orchestra.run.smithi082.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 239 kB/s | 32 kB 00:00 2024-03-29T11:17:49.087 INFO:teuthology.orchestra.run.smithi082.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 533 kB/s | 17 kB 00:00 2024-03-29T11:17:49.129 INFO:teuthology.orchestra.run.smithi082.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 844 kB/s | 34 kB 00:00 2024-03-29T11:17:49.179 INFO:teuthology.orchestra.run.smithi082.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-29T11:17:49.182 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:17:49.182 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:17:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-29T11:17:49.182 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:17:49.182 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:17:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:17:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:49.183 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:17:49.183 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:17:49.183 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-03-29T11:17:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:49.183 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 747 k 2024-03-29T11:17:49.183 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 754 k 2024-03-29T11:17:49.183 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:17:49.355 INFO:teuthology.orchestra.run.smithi082.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-29T11:17:49.422 INFO:teuthology.orchestra.run.smithi082.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-29T11:17:49.447 INFO:teuthology.orchestra.run.smithi082.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T11:17:49.789 INFO:teuthology.orchestra.run.smithi082.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-29T11:17:50.091 INFO:teuthology.orchestra.run.smithi067.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 824 kB/s | 747 kB 00:00 2024-03-29T11:17:50.091 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:50.091 INFO:teuthology.orchestra.run.smithi067.stdout:Total 823 kB/s | 747 kB 00:00 2024-03-29T11:17:50.092 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:17:50.097 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:17:50.098 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:17:50.104 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:17:50.104 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:17:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:17:50.614 INFO:teuthology.orchestra.run.smithi082.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 237 kB/s | 195 kB 00:00 2024-03-29T11:17:50.641 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:50.740 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:50.872 INFO:teuthology.orchestra.run.smithi082.stdout:(10/12): ceph-mgr-diskprediction-local-19.0.0-1 3.8 MB/s | 7.4 MB 00:01 2024-03-29T11:17:51.023 INFO:teuthology.orchestra.run.smithi082.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 601 kB/s | 245 kB 00:00 2024-03-29T11:17:51.256 INFO:teuthology.orchestra.run.smithi082.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 444 kB/s | 803 kB 00:01 2024-03-29T11:17:51.259 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:51.260 INFO:teuthology.orchestra.run.smithi082.stdout:Total 14 MB/s | 37 MB 00:02 2024-03-29T11:17:51.284 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:51.303 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:17:51.329 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:17:51.329 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:17:51.579 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:51.579 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:51.579 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:17:51.579 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:51.579 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:51.580 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:17:51.688 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:17:51.689 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:17:51.765 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:17:52.338 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:17:52.385 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:17:52.454 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:17:52.485 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:17:52.493 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:17:52.495 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:17:52.495 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-03-29T11:17:52.495 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:17:52.495 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:17:52.495 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:17:52.495 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:52.495 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:17:52.495 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:17:52.495 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-03-29T11:17:52.495 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:52.495 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 145 k 2024-03-29T11:17:52.496 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 436 k 2024-03-29T11:17:52.496 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:17:52.578 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:52.613 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:17:52.790 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:52.846 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:52.922 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 220 kB/s | 145 kB 00:00 2024-03-29T11:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:Total 220 kB/s | 145 kB 00:00 2024-03-29T11:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:17:53.165 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:17:53.229 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:17:53.230 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:17:53.374 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:17:53.423 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:53.452 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:53.453 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:53.453 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:17:53.453 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:53.573 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:17:53.684 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:17:54.453 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:54.453 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:54.453 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:17:54.453 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:54.453 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:54.454 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:17:54.686 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr 2024-03-29T11:17:54.936 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:17:55.258 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:17:55.358 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:55.414 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:17:55.415 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-29T11:17:55.415 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:17:55.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:17:55.961 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:17:55.988 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:17:56.047 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:17:56.203 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:17:56.204 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================== 2024-03-29T11:17:56.204 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-29T11:17:56.204 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================== 2024-03-29T11:17:56.205 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:17:56.205 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:17:56.205 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-29T11:17:56.205 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:17:56.205 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:17:56.205 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:17:56.205 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:17:56.205 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:17:56.205 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:17:56.205 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:17:56.205 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:17:56.205 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:17:56.205 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:17:56.205 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:17:56.206 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:56.206 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:17:56.206 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================== 2024-03-29T11:17:56.206 INFO:teuthology.orchestra.run.smithi067.stdout:Install 12 Packages 2024-03-29T11:17:56.206 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:17:56.206 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 6.9 M 2024-03-29T11:17:56.206 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 26 M 2024-03-29T11:17:56.207 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:17:56.792 INFO:teuthology.orchestra.run.smithi067.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 80 kB/s | 15 kB 00:00 2024-03-29T11:17:56.892 INFO:teuthology.orchestra.run.smithi067.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 108 kB/s | 31 kB 00:00 2024-03-29T11:17:56.959 INFO:teuthology.orchestra.run.smithi067.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 5.0 MB/s | 1.8 MB 00:00 2024-03-29T11:17:57.001 INFO:teuthology.orchestra.run.smithi067.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-29T11:17:57.026 INFO:teuthology.orchestra.run.smithi067.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-29T11:17:57.168 INFO:teuthology.orchestra.run.smithi067.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-03-29T11:17:57.327 INFO:teuthology.orchestra.run.smithi067.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.3 MB/s | 863 kB 00:00 2024-03-29T11:17:57.352 INFO:teuthology.orchestra.run.smithi067.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-03-29T11:17:57.410 INFO:teuthology.orchestra.run.smithi067.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 8.4 MB/s | 2.0 MB 00:00 2024-03-29T11:17:57.436 INFO:teuthology.orchestra.run.smithi067.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.3 MB/s | 144 kB 00:00 2024-03-29T11:17:57.461 INFO:teuthology.orchestra.run.smithi067.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 283 kB/s | 31 kB 00:00 2024-03-29T11:17:57.478 INFO:teuthology.orchestra.run.smithi067.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-29T11:17:57.482 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:57.482 INFO:teuthology.orchestra.run.smithi067.stdout:Total 5.4 MB/s | 6.9 MB 00:01 2024-03-29T11:17:57.500 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:17:57.517 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:17:57.518 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:17:57.699 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:17:57.699 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:17:58.220 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:17:58.361 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:17:58.391 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:17:58.427 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:17:58.427 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:17:58.427 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:17:58.427 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:58.427 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:17:58.427 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:58.428 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:58.428 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:17:58.428 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:17:58.428 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:17:58.428 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:17:58.428 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:17:58.494 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:17:58.628 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:58.822 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:17:58.912 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:17:58.912 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:58.912 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:17:58.912 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:58.912 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:17:58.912 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:17:58.912 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:17:58.912 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:17:58.912 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:17:58.912 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:17:58.913 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:17:58.913 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:17:58.913 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:17:58.913 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:17:58.913 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:17:58.913 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:17:58.913 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:17:59.067 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:17:59.126 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:59.329 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:17:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:17:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:17:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:17:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:17:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:17:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:17:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:17:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:17:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:17:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:17:59.903 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:00.056 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:00.057 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================== 2024-03-29T11:18:00.057 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T11:18:00.057 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================== 2024-03-29T11:18:00.057 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:18:00.057 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:18:00.057 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T11:18:00.057 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:18:00.057 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:18:00.057 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:18:00.057 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:18:00.058 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:18:00.058 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:18:00.058 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:18:00.058 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:18:00.058 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:18:00.058 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:18:00.058 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:18:00.058 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:18:00.058 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:00.058 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:18:00.058 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================== 2024-03-29T11:18:00.058 INFO:teuthology.orchestra.run.smithi082.stdout:Install 13 Packages 2024-03-29T11:18:00.058 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:00.059 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 2.0 M 2024-03-29T11:18:00.059 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 26 M 2024-03-29T11:18:00.059 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:18:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:18:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:00.235 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:18:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:18:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:00.235 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:18:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:18:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:18:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:18:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:18:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:18:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:18:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:18:00.236 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:00.236 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:00.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:18:00.529 INFO:teuthology.orchestra.run.smithi082.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 211 kB/s | 24 kB 00:00 2024-03-29T11:18:00.546 INFO:teuthology.orchestra.run.smithi082.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 136 kB/s | 18 kB 00:00 2024-03-29T11:18:00.588 INFO:teuthology.orchestra.run.smithi082.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.0 MB/s | 174 kB 00:00 2024-03-29T11:18:00.613 INFO:teuthology.orchestra.run.smithi082.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:18:00.638 INFO:teuthology.orchestra.run.smithi082.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:18:00.664 INFO:teuthology.orchestra.run.smithi082.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-29T11:18:00.689 INFO:teuthology.orchestra.run.smithi082.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 189 kB/s | 52 kB 00:00 2024-03-29T11:18:00.897 INFO:teuthology.orchestra.run.smithi082.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-03-29T11:18:00.922 INFO:teuthology.orchestra.run.smithi082.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-03-29T11:18:01.043 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:01.131 INFO:teuthology.orchestra.run.smithi082.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 284 kB/s | 59 kB 00:00 2024-03-29T11:18:01.165 INFO:teuthology.orchestra.run.smithi082.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 426 kB/s | 203 kB 00:00 2024-03-29T11:18:01.204 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:01.205 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-03-29T11:18:01.205 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-29T11:18:01.205 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-03-29T11:18:01.205 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:18:01.205 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:18:01.205 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-29T11:18:01.205 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:18:01.205 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:18:01.206 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:18:01.206 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:18:01.206 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:18:01.206 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:18:01.206 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:18:01.206 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:18:01.206 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:18:01.206 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:18:01.206 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:18:01.206 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:01.206 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:18:01.207 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-03-29T11:18:01.207 INFO:teuthology.orchestra.run.smithi067.stdout:Install 12 Packages 2024-03-29T11:18:01.207 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:01.207 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 37 M 2024-03-29T11:18:01.207 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 209 M 2024-03-29T11:18:01.208 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:18:01.298 INFO:teuthology.orchestra.run.smithi082.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 537 kB/s | 90 kB 00:00 2024-03-29T11:18:01.440 INFO:teuthology.orchestra.run.smithi082.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-03-29T11:18:01.442 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:01.443 INFO:teuthology.orchestra.run.smithi082.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-29T11:18:01.443 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:18:01.457 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:18:01.457 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:18:01.549 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:18:01.549 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:18:01.758 INFO:teuthology.orchestra.run.smithi067.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 197 kB/s | 32 kB 00:00 2024-03-29T11:18:01.792 INFO:teuthology.orchestra.run.smithi067.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 534 kB/s | 17 kB 00:00 2024-03-29T11:18:01.834 INFO:teuthology.orchestra.run.smithi067.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 824 kB/s | 34 kB 00:00 2024-03-29T11:18:01.934 INFO:teuthology.orchestra.run.smithi067.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.9 MB/s | 3.0 MB 00:00 2024-03-29T11:18:02.035 INFO:teuthology.orchestra.run.smithi067.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2024-03-29T11:18:02.058 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:18:02.127 INFO:teuthology.orchestra.run.smithi067.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-29T11:18:02.152 INFO:teuthology.orchestra.run.smithi067.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.8 MB/s | 217 kB 00:00 2024-03-29T11:18:02.221 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:18:02.283 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:18:02.327 INFO:teuthology.orchestra.run.smithi067.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-29T11:18:02.338 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:18:02.384 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:18:02.425 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:18:02.478 INFO:teuthology.orchestra.run.smithi067.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-29T11:18:02.503 INFO:teuthology.orchestra.run.smithi067.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-29T11:18:02.520 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:18:02.620 INFO:teuthology.orchestra.run.smithi067.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-29T11:18:02.628 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:18:02.677 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:18:03.399 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.0 MB/s | 7.4 MB 00:01 2024-03-29T11:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-29T11:18:03.440 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:18:03.474 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:18:03.523 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:18:03.586 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:03.880 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:18:03.880 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:18:04.380 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:04.380 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:18:04.380 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:18:04.380 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:18:04.380 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:18:04.381 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:18:04.381 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:18:04.381 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:18:04.381 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:18:04.381 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:18:04.381 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:18:04.381 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:18:04.381 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:18:04.617 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:18:04.693 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:18:04.732 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:18:04.740 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:18:04.740 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:18:04.741 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:04.742 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:04.830 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:04.860 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:18:04.997 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:18:05.046 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:05.076 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:05.160 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:18:05.566 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:00 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:05.717 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:05.718 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================== 2024-03-29T11:18:05.718 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T11:18:05.718 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================== 2024-03-29T11:18:05.718 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:18:05.718 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:18:05.719 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T11:18:05.719 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:18:05.719 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:18:05.719 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:18:05.719 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:18:05.719 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:18:05.719 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:05.719 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:18:05.719 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================== 2024-03-29T11:18:05.719 INFO:teuthology.orchestra.run.smithi082.stdout:Install 6 Packages 2024-03-29T11:18:05.719 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:05.720 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 6.8 M 2024-03-29T11:18:05.720 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 33 M 2024-03-29T11:18:05.720 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:18:05.866 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:18:05.983 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:18:06.275 INFO:teuthology.orchestra.run.smithi082.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-03-29T11:18:06.334 INFO:teuthology.orchestra.run.smithi082.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 486 kB/s | 138 kB 00:00 2024-03-29T11:18:06.476 INFO:teuthology.orchestra.run.smithi082.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-29T11:18:06.751 INFO:teuthology.orchestra.run.smithi082.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 139 kB/s | 58 kB 00:00 2024-03-29T11:18:06.826 INFO:teuthology.orchestra.run.smithi082.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 146 kB/s | 51 kB 00:00 2024-03-29T11:18:06.860 INFO:teuthology.orchestra.run.smithi082.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 942 kB/s | 548 kB 00:00 2024-03-29T11:18:06.861 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:06.862 INFO:teuthology.orchestra.run.smithi082.stdout:Total 5.9 MB/s | 6.8 MB 00:01 2024-03-29T11:18:06.874 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:18:06.884 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:18:06.884 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:18:06.972 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:18:06.972 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:18:07.277 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:18:07.279 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:18:07.435 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:18:07.486 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:18:08.000 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:18:08.080 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:18:08.142 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:18:08.182 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:08.363 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:18:08.391 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:08.972 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:08.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:18:08.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:18:08.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:18:08.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:18:08.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:18:09.262 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:18:09.262 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:09.262 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:18:09.262 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:09.262 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:18:09.262 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:18:09.262 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:18:09.262 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:18:09.262 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:18:09.263 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:09.263 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:09.513 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-fuse 2024-03-29T11:18:10.094 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:05 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:10.256 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:10.257 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:10.257 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T11:18:10.257 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:10.257 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:18:10.257 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:18:10.257 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T11:18:10.257 INFO:teuthology.orchestra.run.smithi082.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:18:10.257 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:10.257 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:18:10.257 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:10.258 INFO:teuthology.orchestra.run.smithi082.stdout:Install 2 Packages 2024-03-29T11:18:10.258 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:10.258 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 964 k 2024-03-29T11:18:10.258 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 2.8 M 2024-03-29T11:18:10.258 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:18:10.388 INFO:teuthology.orchestra.run.smithi082.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 606 kB/s | 79 kB 00:00 2024-03-29T11:18:10.868 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:10.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:18:10.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:18:10.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:10.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:18:10.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:10.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:10.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:18:10.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:18:10.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:18:10.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:18:10.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:18:11.097 INFO:teuthology.orchestra.run.smithi082.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.0 MB/s | 885 kB 00:00 2024-03-29T11:18:11.098 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:11.099 INFO:teuthology.orchestra.run.smithi082.stdout:Total 1.1 MB/s | 964 kB 00:00 2024-03-29T11:18:11.099 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:18:11.121 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:18:11.121 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:18:11.197 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:18:11.198 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:18:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:18:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:11.263 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:18:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:18:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:18:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:18:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:18:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:18:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:18:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:18:11.263 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:18:11.264 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:11.264 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:11.264 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:18:11.264 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:11.264 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:11.385 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:18:11.522 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:11.596 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:11.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:18:12.257 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:09 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:12.269 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:12.270 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:12.415 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:12.416 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-03-29T11:18:12.416 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-29T11:18:12.416 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-03-29T11:18:12.416 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:18:12.416 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:18:12.416 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-29T11:18:12.417 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:18:12.417 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:18:12.417 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:18:12.417 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:18:12.417 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:18:12.417 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:18:12.417 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:18:12.417 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:18:12.417 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:18:12.417 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:18:12.417 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:18:12.418 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:18:12.418 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:12.418 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:18:12.418 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-03-29T11:18:12.418 INFO:teuthology.orchestra.run.smithi067.stdout:Install 13 Packages 2024-03-29T11:18:12.418 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:12.419 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 2.0 M 2024-03-29T11:18:12.419 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 26 M 2024-03-29T11:18:12.419 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:18:12.552 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:12.552 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:12.552 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:18:12.552 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:12.552 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:12.552 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:12.657 INFO:teuthology.orchestra.run.smithi067.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 157 kB/s | 18 kB 00:00 2024-03-29T11:18:12.674 INFO:teuthology.orchestra.run.smithi067.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 184 kB/s | 24 kB 00:00 2024-03-29T11:18:12.724 INFO:teuthology.orchestra.run.smithi067.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-29T11:18:12.750 INFO:teuthology.orchestra.run.smithi067.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:18:12.775 INFO:teuthology.orchestra.run.smithi067.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:18:12.801 INFO:teuthology.orchestra.run.smithi067.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-29T11:18:12.851 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-volume 2024-03-29T11:18:12.900 INFO:teuthology.orchestra.run.smithi067.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 146 kB/s | 52 kB 00:00 2024-03-29T11:18:12.934 INFO:teuthology.orchestra.run.smithi067.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 203 kB/s | 32 kB 00:00 2024-03-29T11:18:12.968 INFO:teuthology.orchestra.run.smithi067.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 90 kB/s | 15 kB 00:00 2024-03-29T11:18:13.018 INFO:teuthology.orchestra.run.smithi067.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.7 MB/s | 203 kB 00:00 2024-03-29T11:18:13.043 INFO:teuthology.orchestra.run.smithi067.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 787 kB/s | 59 kB 00:00 2024-03-29T11:18:13.085 INFO:teuthology.orchestra.run.smithi067.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-29T11:18:13.269 INFO:teuthology.orchestra.run.smithi067.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-03-29T11:18:13.272 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:13.272 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-03-29T11:18:13.273 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:18:13.287 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:18:13.287 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:18:13.384 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:18:13.384 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:18:13.429 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:08 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:13.585 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:13.586 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T11:18:13.586 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:13.586 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:18:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:13.586 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T11:18:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:13.586 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:13.587 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:18:13.587 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:13.587 INFO:teuthology.orchestra.run.smithi082.stdout:Install 2 Packages 2024-03-29T11:18:13.587 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:13.587 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 332 k 2024-03-29T11:18:13.587 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 1.5 M 2024-03-29T11:18:13.587 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:18:13.760 INFO:teuthology.orchestra.run.smithi082.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 399 kB/s | 69 kB 00:00 2024-03-29T11:18:13.911 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:18:14.026 INFO:teuthology.orchestra.run.smithi082.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 601 kB/s | 264 kB 00:00 2024-03-29T11:18:14.027 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:14.028 INFO:teuthology.orchestra.run.smithi082.stdout:Total 753 kB/s | 332 kB 00:00 2024-03-29T11:18:14.028 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:18:14.036 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:18:14.036 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:18:14.071 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:18:14.071 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:18:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:18:14.146 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:18:14.211 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:18:14.220 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:18:14.246 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:18:14.288 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:18:14.390 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:18:14.402 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:14.438 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:14.454 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:14.454 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:14.504 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:18:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:18:15.231 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:15.348 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:18:15.394 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:18:15.428 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:18:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:18:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:15.589 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:15.589 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:15.589 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:18:15.589 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:15.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:15.589 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:15.589 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:15.934 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librados-devel 2024-03-29T11:18:16.281 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:16.281 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:18:16.281 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:18:16.281 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:18:16.282 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:18:16.282 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:18:16.282 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:18:16.282 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:18:16.282 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:18:16.282 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:18:16.282 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:18:16.282 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:18:16.282 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:18:16.508 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:11 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:16.659 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:16.659 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:16.659 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-03-29T11:18:16.660 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:16.660 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:18:16.660 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:18:16.660 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:16.660 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:18:16.660 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:16.660 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-03-29T11:18:16.660 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:16.661 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 127 k 2024-03-29T11:18:16.661 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 456 k 2024-03-29T11:18:16.661 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:18:16.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:18:17.141 INFO:teuthology.orchestra.run.smithi082.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 264 kB/s | 127 kB 00:00 2024-03-29T11:18:17.141 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:17.142 INFO:teuthology.orchestra.run.smithi082.stdout:Total 263 kB/s | 127 kB 00:00 2024-03-29T11:18:17.142 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:18:17.147 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:18:17.147 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:18:17.203 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:18:17.203 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:18:17.346 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:18:17.471 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:14 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:17.473 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:17.630 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:17.631 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-03-29T11:18:17.631 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-29T11:18:17.631 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-03-29T11:18:17.631 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:18:17.631 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:18:17.632 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-29T11:18:17.632 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:18:17.632 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:18:17.632 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:18:17.632 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:18:17.632 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:18:17.632 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:17.633 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:18:17.633 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-03-29T11:18:17.633 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-03-29T11:18:17.633 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:17.634 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 6.8 M 2024-03-29T11:18:17.634 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 33 M 2024-03-29T11:18:17.634 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:18:17.934 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:18.208 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:18:18.208 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:18.209 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:18.209 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:18.295 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T11:18:18.312 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 493 kB/s | 138 kB 00:00 2024-03-29T11:18:18.337 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 813 kB/s | 248 kB 00:00 2024-03-29T11:18:18.443 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install libcephfs2 2024-03-29T11:18:18.637 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 177 kB/s | 58 kB 00:00 2024-03-29T11:18:18.662 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 157 kB/s | 51 kB 00:00 2024-03-29T11:18:18.887 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 925 kB/s | 548 kB 00:00 2024-03-29T11:18:18.888 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:18.889 INFO:teuthology.orchestra.run.smithi067.stdout:Total 5.4 MB/s | 6.8 MB 00:01 2024-03-29T11:18:18.906 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:18:18.915 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:18:18.915 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:18:19.010 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:18:19.010 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:18:19.021 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:14 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:19.122 INFO:teuthology.orchestra.run.smithi082.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:19.178 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:19.179 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T11:18:19.180 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:19.250 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:19.318 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:18:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:18:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:18:19.817 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:14 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:19.967 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:19.968 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:19.968 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-03-29T11:18:19.968 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:19.968 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:18:19.968 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:18:19.969 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:19.969 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:18:19.969 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:19.969 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-03-29T11:18:19.969 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:19.969 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 31 k 2024-03-29T11:18:19.969 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 140 k 2024-03-29T11:18:19.969 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:18:20.100 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:18:20.194 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:18:20.228 INFO:teuthology.orchestra.run.smithi082.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 120 kB/s | 31 kB 00:00 2024-03-29T11:18:20.228 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:20.229 INFO:teuthology.orchestra.run.smithi082.stdout:Total 120 kB/s | 31 kB 00:00 2024-03-29T11:18:20.229 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:18:20.232 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:18:20.232 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:18:20.261 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:18:20.261 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:18:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:18:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:20.402 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:18:20.511 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:20.881 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:18:21.132 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:21.132 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:21.132 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:18:21.132 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:21.132 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:21.132 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:21.320 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librados2 2024-03-29T11:18:21.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:18:21.409 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:21.409 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:18:21.409 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:21.409 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:18:21.409 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:18:21.409 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:18:21.409 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:18:21.409 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:18:21.410 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:21.410 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:21.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-fuse 2024-03-29T11:18:21.887 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:16 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:21.984 INFO:teuthology.orchestra.run.smithi082.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:22.037 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:22.038 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T11:18:22.038 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:22.087 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librbd1 2024-03-29T11:18:22.190 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:22.349 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:22.350 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:22.350 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-29T11:18:22.350 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:22.350 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:18:22.350 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:18:22.350 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-29T11:18:22.350 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:18:22.350 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:22.351 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:18:22.351 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:22.351 INFO:teuthology.orchestra.run.smithi067.stdout:Install 2 Packages 2024-03-29T11:18:22.351 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:22.351 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 964 k 2024-03-29T11:18:22.351 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 2.8 M 2024-03-29T11:18:22.351 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:18:22.464 INFO:teuthology.orchestra.run.smithi067.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 696 kB/s | 79 kB 00:00 2024-03-29T11:18:22.650 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:17 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:22.750 INFO:teuthology.orchestra.run.smithi082.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:22.802 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:22.803 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T11:18:22.803 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:22.856 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rados 2024-03-29T11:18:23.097 INFO:teuthology.orchestra.run.smithi067.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 885 kB 00:00 2024-03-29T11:18:23.099 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:23.099 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.3 MB/s | 964 kB 00:00 2024-03-29T11:18:23.099 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:18:23.120 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:18:23.120 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:18:23.197 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:18:23.197 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:18:23.381 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:18:23.419 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:23.517 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:23.519 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:23.568 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:23.569 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T11:18:23.570 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:23.591 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:23.621 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rgw 2024-03-29T11:18:24.182 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:24.267 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:24.267 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:24.280 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:24.332 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T11:18:24.333 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:24.389 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-cephfs 2024-03-29T11:18:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:24.568 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:24.568 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:18:24.569 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:24.569 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:24.569 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:24.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-volume 2024-03-29T11:18:24.960 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:25.058 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:25.110 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:25.111 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T11:18:25.111 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:25.165 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rbd 2024-03-29T11:18:25.431 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:22 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:25.590 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:25.590 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-03-29T11:18:25.590 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:25.590 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:18:25.590 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:25.590 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-03-29T11:18:25.591 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:25.591 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:25.591 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:18:25.591 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:25.591 INFO:teuthology.orchestra.run.smithi067.stdout:Install 2 Packages 2024-03-29T11:18:25.591 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:25.592 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 332 k 2024-03-29T11:18:25.592 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 1.5 M 2024-03-29T11:18:25.592 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:18:25.725 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:25.727 INFO:teuthology.orchestra.run.smithi067.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 509 kB/s | 69 kB 00:00 2024-03-29T11:18:25.824 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:25.876 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:25.877 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T11:18:25.877 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:25.931 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-fuse 2024-03-29T11:18:25.986 INFO:teuthology.orchestra.run.smithi067.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 670 kB/s | 264 kB 00:00 2024-03-29T11:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout:Total 839 kB/s | 332 kB 00:00 2024-03-29T11:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:18:26.189 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:18:26.376 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:26.413 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:26.429 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:26.429 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:26.429 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:26.429 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:26.491 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:26.641 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:26.642 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:26.642 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:26.642 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:26.642 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:18:26.642 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:18:26.642 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:26.642 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:18:26.642 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:26.643 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-03-29T11:18:26.643 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:26.643 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 85 k 2024-03-29T11:18:26.643 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 231 k 2024-03-29T11:18:26.643 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:18:27.047 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 211 kB/s | 85 kB 00:00 2024-03-29T11:18:27.047 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:27.047 INFO:teuthology.orchestra.run.smithi082.stdout:Total 210 kB/s | 85 kB 00:00 2024-03-29T11:18:27.048 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:18:27.055 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:18:27.055 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:18:27.099 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:27.113 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:18:27.113 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:18:27.250 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:18:27.355 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:27.441 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:27.441 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:27.441 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:18:27.441 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:27.441 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:27.441 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:27.441 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:27.751 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:27.758 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados-devel 2024-03-29T11:18:28.039 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:28.040 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:28.040 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:18:28.040 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:28.040 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:28.040 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:28.313 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-mirror 2024-03-29T11:18:28.339 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:25 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:28.495 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:28.496 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:28.496 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-03-29T11:18:28.496 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:28.496 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:18:28.496 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:18:28.496 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:28.496 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:18:28.496 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:28.496 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-03-29T11:18:28.496 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:28.497 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 127 k 2024-03-29T11:18:28.497 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 456 k 2024-03-29T11:18:28.497 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:18:28.716 INFO:teuthology.orchestra.run.smithi067.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 577 kB/s | 127 kB 00:00 2024-03-29T11:18:28.717 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:28.717 INFO:teuthology.orchestra.run.smithi067.stdout:Total 574 kB/s | 127 kB 00:00 2024-03-29T11:18:28.717 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:18:28.723 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:18:28.723 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:18:28.782 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:18:28.782 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:18:28.887 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:28.944 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:18:29.042 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:29.043 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:29.043 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-03-29T11:18:29.043 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:29.043 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:18:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:29.043 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:29.043 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:18:29.043 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:29.044 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-03-29T11:18:29.044 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:29.044 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 3.1 M 2024-03-29T11:18:29.044 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 13 M 2024-03-29T11:18:29.044 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:18:29.451 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:29.708 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:29.708 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:29.708 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:18:29.708 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:29.708 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:29.708 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:29.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs2 2024-03-29T11:18:30.504 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:30.606 INFO:teuthology.orchestra.run.smithi067.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:30.663 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:30.664 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-29T11:18:30.664 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:30.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:31.241 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.4 MB/s | 3.1 MB 00:02 2024-03-29T11:18:31.242 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:31.242 INFO:teuthology.orchestra.run.smithi082.stdout:Total 1.4 MB/s | 3.1 MB 00:02 2024-03-29T11:18:31.243 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:18:31.253 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:18:31.253 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:28 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:31.329 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:18:31.330 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:18:31.459 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:31.460 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:31.460 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-03-29T11:18:31.460 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:31.460 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:18:31.460 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:18:31.460 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:31.460 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:18:31.460 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:31.460 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-03-29T11:18:31.461 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:31.461 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 31 k 2024-03-29T11:18:31.461 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 140 k 2024-03-29T11:18:31.461 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:18:31.770 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:18:31.838 INFO:teuthology.orchestra.run.smithi067.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 83 kB/s | 31 kB 00:00 2024-03-29T11:18:31.838 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:31.838 INFO:teuthology.orchestra.run.smithi067.stdout:Total 82 kB/s | 31 kB 00:00 2024-03-29T11:18:31.838 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:18:31.842 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:18:31.842 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:18:31.850 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:31.872 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:18:31.872 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:18:31.875 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:31.875 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:31.875 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:18:31.875 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:31.875 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:31.875 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:31.985 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:18:32.080 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:32.460 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:32.722 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:18:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:32.722 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:32.885 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados2 2024-03-29T11:18:33.005 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:33.005 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:33.006 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:18:33.006 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:33.006 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:33.006 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:33.267 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-nbd 2024-03-29T11:18:33.465 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:30 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:33.565 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:33.619 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:33.620 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-29T11:18:33.620 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:33.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librbd1 2024-03-29T11:18:33.841 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:02:28 ago on Fri 29 Mar 2024 11:16:05 AM UTC. 2024-03-29T11:18:33.991 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T11:18:33.991 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:33.991 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:33.992 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:33.992 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T11:18:33.992 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:18:33.992 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:33.992 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T11:18:33.992 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T11:18:33.992 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-03-29T11:18:33.992 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:33.992 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 171 k 2024-03-29T11:18:33.992 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 494 k 2024-03-29T11:18:33.992 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T11:18:34.237 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:31 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:34.237 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 703 kB/s | 171 kB 00:00 2024-03-29T11:18:34.237 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:34.237 INFO:teuthology.orchestra.run.smithi082.stdout:Total 699 kB/s | 171 kB 00:00 2024-03-29T11:18:34.237 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T11:18:34.245 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T11:18:34.245 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T11:18:34.301 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T11:18:34.301 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T11:18:34.335 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:34.390 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:34.391 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-29T11:18:34.392 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:34.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rados 2024-03-29T11:18:34.450 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T11:18:34.561 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:34.961 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:35.021 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:32 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:35.123 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:35.177 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:35.178 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-29T11:18:35.178 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:35.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rgw 2024-03-29T11:18:35.257 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:35.258 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:35.258 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T11:18:35.258 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:35.258 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:35.258 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T11:18:35.537 DEBUG:teuthology.parallel:result is None 2024-03-29T11:18:35.801 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:32 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:35.903 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:35.959 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:35.960 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-29T11:18:35.960 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:36.009 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-cephfs 2024-03-29T11:18:36.588 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:33 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:36.690 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:36.746 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:36.747 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-29T11:18:36.747 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:36.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rbd 2024-03-29T11:18:37.371 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:34 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:37.472 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:37.528 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:37.528 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-03-29T11:18:37.529 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:37.575 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-fuse 2024-03-29T11:18:38.150 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:35 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:38.305 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:38.305 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:38.306 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:38.306 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:18:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:18:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:38.306 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:18:38.306 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:38.306 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-03-29T11:18:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:38.306 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 85 k 2024-03-29T11:18:38.306 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 231 k 2024-03-29T11:18:38.306 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:18:38.574 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 319 kB/s | 85 kB 00:00 2024-03-29T11:18:38.575 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:38.575 INFO:teuthology.orchestra.run.smithi067.stdout:Total 317 kB/s | 85 kB 00:00 2024-03-29T11:18:38.575 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:18:38.582 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:18:38.582 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:18:38.639 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:18:38.639 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:18:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:18:38.883 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:39.261 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:39.548 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:18:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:39.548 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:39.548 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:39.784 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-mirror 2024-03-29T11:18:40.364 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:37 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:40.519 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:40.520 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:40.520 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-03-29T11:18:40.520 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:40.520 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:18:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:40.521 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:18:40.521 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:40.521 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-03-29T11:18:40.521 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:40.521 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 3.1 M 2024-03-29T11:18:40.521 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 13 M 2024-03-29T11:18:40.521 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:18:41.184 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 4.7 MB/s | 3.1 MB 00:00 2024-03-29T11:18:41.185 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:41.185 INFO:teuthology.orchestra.run.smithi067.stdout:Total 4.6 MB/s | 3.1 MB 00:00 2024-03-29T11:18:41.185 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:18:41.195 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:18:41.195 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:18:41.272 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:18:41.273 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:18:41.723 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:18:41.785 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:41.809 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:41.809 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:41.810 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:18:41.810 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:41.810 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:41.810 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:42.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:42.889 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:42.889 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:18:42.890 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:42.890 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:42.890 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:43.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-nbd 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:40 ago on Fri 29 Mar 2024 11:16:03 AM UTC. 2024-03-29T11:18:43.854 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-03-29T11:18:43.855 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:43.855 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:43.855 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-03-29T11:18:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:18:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:43.855 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-03-29T11:18:43.856 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-03-29T11:18:43.856 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-03-29T11:18:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:43.856 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 171 k 2024-03-29T11:18:43.856 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 494 k 2024-03-29T11:18:43.856 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-03-29T11:18:44.204 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 492 kB/s | 171 kB 00:00 2024-03-29T11:18:44.205 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:44.205 INFO:teuthology.orchestra.run.smithi067.stdout:Total 490 kB/s | 171 kB 00:00 2024-03-29T11:18:44.205 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-03-29T11:18:44.214 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-03-29T11:18:44.214 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-03-29T11:18:44.274 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-03-29T11:18:44.274 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-03-29T11:18:44.427 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-03-29T11:18:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:45.273 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-03-29T11:18:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:45.273 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-03-29T11:18:45.515 DEBUG:teuthology.parallel:result is None 2024-03-29T11:18:45.516 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:18:45.653 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:18:45.674 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:45.674 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:45.675 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:18:45.676 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:18:45.818 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:18:45.843 INFO:teuthology.orchestra.run.smithi082.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:45.843 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:45.843 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:18:45.845 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:18:45.845 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:18:45.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:18:45.874 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:18:45.874 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:18:45.917 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:18:45.917 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:18:45.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:18:45.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:18:46.012 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:18:46.012 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:18:46.041 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:18:46.110 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:18:46.110 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:18:46.110 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:18:46.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:18:46.212 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:18:46.212 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:18:46.240 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:18:46.307 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:18:46.307 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:18:46.307 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:18:46.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:18:46.401 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:18:46.401 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:18:46.429 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:18:46.496 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:18:46.614 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:18:46.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2024-03-29T11:18:46.616 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /var/log/ceph 2024-03-29T11:18:46.643 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:18:46.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:18:46.685 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:18:46.712 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:18:46.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:18:46.752 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:18:46.797 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:18:46.798 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:18:46.798 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622693', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOWEyAUDB6nKarlTJ77qio03BlccevtgBeAzfG88npriw1GXRuGLuaygY7KdT1Cok+NjmfiQsE7OoEF4dEyVwKc=', 'smithi082.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPnLy+p2JFVioUa8/WLiHzMs21gwvt5OFzRAMN/H+2A0+xE4eK/hll2PJ4DvI6L8aWJb9ENYFvMXZgSywlqtsw0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:18:46.798 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:18:46.820 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:18:46.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:18:46.875 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:18:46.902 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:18:46.902 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:18:46.956 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:18:46.956 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-03-29T11:18:47.010 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:18:47.010 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:47.010 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:18:47.010 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:47.010 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:47.010 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-29 11:18:42.079537723 +0000 2024-03-29T11:18:47.010 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-29 11:15:18.127661828 +0000 2024-03-29T11:18:47.010 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-29 11:15:18.127661828 +0000 2024-03-29T11:18:47.011 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-29 11:15:18.127661828 +0000 2024-03-29T11:18:47.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:18:47.076 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-29T11:18:47.076 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-29T11:18:47.076 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000251898 s, 2.0 MB/s 2024-03-29T11:18:47.077 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:18:47.133 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-03-29T11:18:47.188 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:18:47.188 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:47.188 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T11:18:47.188 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:47.188 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:47.188 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-29 11:18:42.079537723 +0000 2024-03-29T11:18:47.188 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-29 11:15:18.481654395 +0000 2024-03-29T11:18:47.188 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-29 11:15:18.481654395 +0000 2024-03-29T11:18:47.188 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-29 11:15:18.481654395 +0000 2024-03-29T11:18:47.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:18:47.254 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-29T11:18:47.254 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-29T11:18:47.254 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000176411 s, 2.9 MB/s 2024-03-29T11:18:47.255 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:18:47.310 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-03-29T11:18:47.366 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:18:47.366 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:47.366 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:18:47.366 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:47.366 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:47.366 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-29 11:18:42.080537702 +0000 2024-03-29T11:18:47.367 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-29 11:15:18.823647215 +0000 2024-03-29T11:18:47.367 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-29 11:15:18.823647215 +0000 2024-03-29T11:18:47.367 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-29 11:15:18.823647215 +0000 2024-03-29T11:18:47.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:18:47.431 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-29T11:18:47.432 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-29T11:18:47.432 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000183337 s, 2.8 MB/s 2024-03-29T11:18:47.433 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:18:47.489 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-03-29T11:18:47.544 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:18:47.544 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:47.544 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:18:47.544 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:47.544 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:47.544 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-29 11:18:42.080537702 +0000 2024-03-29T11:18:47.544 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-29 11:15:19.186639594 +0000 2024-03-29T11:18:47.544 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-29 11:15:19.186639594 +0000 2024-03-29T11:18:47.544 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-03-29 11:15:19.186639594 +0000 2024-03-29T11:18:47.544 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:18:47.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-29T11:18:47.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-29T11:18:47.610 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000171164 s, 3.0 MB/s 2024-03-29T11:18:47.611 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:18:47.666 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:47.667 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:18:47.667 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:18:47.685 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:18:47.686 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2024-03-29T11:18:47.743 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:18:47.743 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:47.743 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:18:47.743 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:47.743 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:47.743 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-29 11:18:32.141525883 +0000 2024-03-29T11:18:47.743 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-29 11:15:12.932201322 +0000 2024-03-29T11:18:47.743 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-29 11:15:12.932201322 +0000 2024-03-29T11:18:47.743 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-29 11:15:12.932201322 +0000 2024-03-29T11:18:47.744 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:18:47.811 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-29T11:18:47.811 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-29T11:18:47.811 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000221775 s, 2.3 MB/s 2024-03-29T11:18:47.813 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:18:47.871 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2024-03-29T11:18:47.927 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:18:47.928 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:47.928 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T11:18:47.928 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:47.928 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:47.928 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-29 11:18:32.142525864 +0000 2024-03-29T11:18:47.928 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-29 11:15:13.280194562 +0000 2024-03-29T11:18:47.928 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-29 11:15:13.280194562 +0000 2024-03-29T11:18:47.928 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-29 11:15:13.280194562 +0000 2024-03-29T11:18:47.929 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:18:47.996 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-29T11:18:47.996 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-29T11:18:47.996 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000260943 s, 2.0 MB/s 2024-03-29T11:18:47.997 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:18:48.055 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2024-03-29T11:18:48.113 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:18:48.113 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:48.113 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:18:48.113 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:48.113 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:48.113 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-29 11:18:32.142525864 +0000 2024-03-29T11:18:48.113 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-29 11:15:13.614188074 +0000 2024-03-29T11:18:48.113 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-29 11:15:13.614188074 +0000 2024-03-29T11:18:48.113 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-29 11:15:13.614188074 +0000 2024-03-29T11:18:48.114 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:18:48.182 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-29T11:18:48.182 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-29T11:18:48.182 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000193286 s, 2.6 MB/s 2024-03-29T11:18:48.183 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:18:48.240 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2024-03-29T11:18:48.297 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:18:48.297 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:48.297 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T11:18:48.298 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:48.298 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:48.298 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-29 11:18:32.142525864 +0000 2024-03-29T11:18:48.298 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-29 11:15:14.109178459 +0000 2024-03-29T11:18:48.298 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-29 11:15:14.109178459 +0000 2024-03-29T11:18:48.298 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-29 11:15:14.109178459 +0000 2024-03-29T11:18:48.298 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:18:48.366 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-29T11:18:48.366 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-29T11:18:48.366 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000215577 s, 2.4 MB/s 2024-03-29T11:18:48.367 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:18:48.425 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:48.425 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:18:48.425 INFO:tasks.ceph:Generating config... 2024-03-29T11:18:48.426 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:18:48.426 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:18:48.426 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:18:48.426 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:18:48.427 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:18:48.427 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:18:48.427 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:18:48.427 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:18:48.427 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:18:48.427 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:18:48.427 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:18:48.427 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:18:48.427 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:18:48.427 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:18:48.427 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:18:48.427 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:18:48.427 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:18:48.427 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:18:48.427 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:18:48.428 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:18:48.428 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:18:48.428 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:18:48.428 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:18:48.428 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:18:48.428 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:18:48.428 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:18:48.428 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:18:48.428 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:18:48.428 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:18:48.428 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:18:48.428 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:18:48.428 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:18:48.428 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:18:48.428 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:48.429 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:18:48.429 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:18:48.429 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:18:48.429 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:18:48.429 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:18:48.429 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:18:48.429 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:18:48.429 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:18:48.429 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:18:48.429 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:18:48.429 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:18:48.429 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:18:48.429 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:18:48.429 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:18:48.430 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:18:48.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:18:48.482 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:18:48.484 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:18:48.531 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:48.559 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.67:6789]'), ('mon.c', '[v1:172.21.15.67:6790]'), ('mon.b', '[v1:172.21.15.82:6789]')] 2024-03-29T11:18:48.559 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.67:6789],[v1:172.21.15.67:6790],[v1:172.21.15.82:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:18:48.560 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:18:48.560 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:18:48.614 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.67:6789]' --addv c '[v1:172.21.15.67:6790]' --addv b '[v1:172.21.15.82:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:48.701 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:48.701 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:18:48.701 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = quincy 2024-03-29T11:18:48.701 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2024-03-29T11:18:48.701 INFO:teuthology.orchestra.run.smithi067.stdout:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:18:48.701 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:18:48.701 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:18:48.701 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 17 (quincy) 2024-03-29T11:18:48.701 INFO:teuthology.orchestra.run.smithi067.stdout:election_strategy: 1 2024-03-29T11:18:48.702 INFO:teuthology.orchestra.run.smithi067.stdout:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:18:48.702 INFO:teuthology.orchestra.run.smithi067.stdout:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:18:48.702 INFO:teuthology.orchestra.run.smithi067.stdout:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:18:48.702 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:18:48.703 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:18:48.717 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 312c75bc-59e5-460a-bb56-f686dd06c6fb... 2024-03-29T11:18:48.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:18:48.759 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:18:48.808 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-03-29T11:18:48.808 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-03-29T11:18:48.808 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:18:48.808 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-03-29T11:18:48.808 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.808 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-03-29T11:18:48.808 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.808 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-03-29T11:18:48.808 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.808 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-03-29T11:18:48.809 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-03-29T11:18:48.809 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.809 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-03-29T11:18:48.809 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-03-29T11:18:48.809 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.809 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:18:48.809 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-03-29T11:18:48.809 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.809 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:18:48.809 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:18:48.809 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.810 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-03-29T11:18:48.810 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.810 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-03-29T11:18:48.810 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-03-29T11:18:48.810 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:18:48.810 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:18:48.810 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-03-29T11:18:48.810 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:18:48.810 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-03-29T11:18:48.810 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:18:48.810 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:18:48.811 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-03-29T11:18:48.811 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.811 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:18:48.811 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.811 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-03-29T11:18:48.811 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.811 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-03-29T11:18:48.811 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.811 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-03-29T11:18:48.811 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-03-29T11:18:48.811 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-03-29T11:18:48.811 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "[v1:172.21.15.67:6789],[v1:172.21.15.67:6790],[v1:172.21.15.82:6789]" 2024-03-29T11:18:48.811 INFO:teuthology.orchestra.run.smithi067.stdout: lockdep = True 2024-03-29T11:18:48.811 INFO:teuthology.orchestra.run.smithi067.stdout: mon client directed command retry = 5 2024-03-29T11:18:48.812 INFO:teuthology.orchestra.run.smithi067.stdout: mon election default strategy = 1 2024-03-29T11:18:48.812 INFO:teuthology.orchestra.run.smithi067.stdout: ms bind msgr2 = False 2024-03-29T11:18:48.812 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject socket failures = 5000 2024-03-29T11:18:48.812 INFO:teuthology.orchestra.run.smithi067.stdout: ms type = async 2024-03-29T11:18:48.812 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:18:48.812 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.812 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-03-29T11:18:48.812 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-03-29T11:18:48.812 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.812 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-03-29T11:18:48.812 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-03-29T11:18:48.812 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:18:48.812 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.813 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-03-29T11:18:48.813 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-03-29T11:18:48.813 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.813 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-03-29T11:18:48.813 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-03-29T11:18:48.813 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-03-29T11:18:48.814 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-03-29T11:18:48.814 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.814 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-03-29T11:18:48.814 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-03-29T11:18:48.814 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.815 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:18:48.815 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.815 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-03-29T11:18:48.815 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.815 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-03-29T11:18:48.815 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.816 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:18:48.816 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-03-29T11:18:48.816 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.816 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-03-29T11:18:48.816 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-03-29T11:18:48.816 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-03-29T11:18:48.817 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore compression algorithm = snappy 2024-03-29T11:18:48.817 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore compression mode = aggressive 2024-03-29T11:18:48.817 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-03-29T11:18:48.817 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore zero block detection = True 2024-03-29T11:18:48.817 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 20 2024-03-29T11:18:48.817 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 20 2024-03-29T11:18:48.818 INFO:teuthology.orchestra.run.smithi067.stdout: debug monc = 20 2024-03-29T11:18:48.818 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-29T11:18:48.818 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-03-29T11:18:48.818 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 10 2024-03-29T11:18:48.818 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:48.818 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-03-29T11:18:48.818 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:18:48.818 INFO:teuthology.orchestra.run.smithi067.stdout: osd class default list = * 2024-03-29T11:18:48.819 INFO:teuthology.orchestra.run.smithi067.stdout: osd class load list = * 2024-03-29T11:18:48.819 INFO:teuthology.orchestra.run.smithi067.stdout: osd client watch timeout = 120 2024-03-29T11:18:48.819 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify cached snaps = True 2024-03-29T11:18:48.819 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify missing on start = True 2024-03-29T11:18:48.819 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:18:48.819 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock override recovery settings = True 2024-03-29T11:18:48.819 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-03-29T11:18:48.819 INFO:teuthology.orchestra.run.smithi067.stdout: osd op queue = debug_random 2024-03-29T11:18:48.819 INFO:teuthology.orchestra.run.smithi067.stdout: osd op queue cut off = debug_random 2024-03-29T11:18:48.819 INFO:teuthology.orchestra.run.smithi067.stdout: osd shutdown pgref assert = True 2024-03-29T11:18:48.819 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.819 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-03-29T11:18:48.820 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-29T11:18:48.820 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-03-29T11:18:48.820 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-03-29T11:18:48.820 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-03-29T11:18:48.820 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:48.820 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:48.820 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-03-29T11:18:48.820 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.820 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-03-29T11:18:48.820 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-29T11:18:48.820 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:18:48.821 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:18:48.822 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:18:48.822 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.822 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-03-29T11:18:48.822 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:18:48.822 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.822 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:18:48.822 INFO:teuthology.orchestra.run.smithi067.stdout: mon min osdmap epochs = 50 2024-03-29T11:18:48.822 INFO:teuthology.orchestra.run.smithi067.stdout: mon osdmap full prune interval = 2 2024-03-29T11:18:48.822 INFO:teuthology.orchestra.run.smithi067.stdout: mon osdmap full prune min = 15 2024-03-29T11:18:48.822 INFO:teuthology.orchestra.run.smithi067.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:18:48.822 INFO:teuthology.orchestra.run.smithi067.stdout: mon scrub interval = 300 2024-03-29T11:18:48.822 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on pool no app = False 2024-03-29T11:18:48.822 INFO:teuthology.orchestra.run.smithi067.stdout: paxos service trim min = 10 2024-03-29T11:18:48.822 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:18:48.823 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-03-29T11:18:48.823 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-03-29T11:18:48.823 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-03-29T11:18:48.823 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-03-29T11:18:48.823 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:18:48.823 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:18:48.823 INFO:teuthology.orchestra.run.smithi067.stdout: debug monc = 20 2024-03-29T11:18:48.823 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-29T11:18:48.823 INFO:teuthology.orchestra.run.smithi067.stdout: debug objecter = 20 2024-03-29T11:18:48.823 INFO:teuthology.orchestra.run.smithi067.stdout: debug rados = 20 2024-03-29T11:18:48.823 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-03-29T11:18:48.823 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-03-29T11:18:48.823 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-03-29T11:18:48.824 INFO:teuthology.orchestra.run.smithi082.stdout:[global] 2024-03-29T11:18:48.824 INFO:teuthology.orchestra.run.smithi082.stdout: chdir = "" 2024-03-29T11:18:48.824 INFO:teuthology.orchestra.run.smithi082.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:18:48.824 INFO:teuthology.orchestra.run.smithi082.stdout: auth supported = cephx 2024-03-29T11:18:48.824 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.824 INFO:teuthology.orchestra.run.smithi082.stdout: filestore xattr use omap = true 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: mon clock drift allowed = 1.000 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: osd crush chooseleaf type = 0 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: auth debug = true 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on old message = true 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on bug = true 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: mon pg warn max object skew = 0 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:18:48.825 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.826 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default size = 2 2024-03-29T11:18:48.826 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.826 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow primary affinity = true 2024-03-29T11:18:48.826 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow pg remap = true 2024-03-29T11:18:48.826 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:18:48.826 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:18:48.826 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on no sortbitwise = false 2024-03-29T11:18:48.826 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:18:48.826 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on too few osds = false 2024-03-29T11:18:48.826 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:18:48.826 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:18:48.826 INFO:teuthology.orchestra.run.smithi082.stdout: mon_allow_pool_size_one = true 2024-03-29T11:18:48.826 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.826 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:18:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: osd default data pool replay window = 5 2024-03-29T11:18:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: mon allow pool delete = true 2024-03-29T11:18:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: mon cluster log file level = debug 2024-03-29T11:18:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: debug asserts on shutdown = true 2024-03-29T11:18:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: mon health detail to clog = false 2024-03-29T11:18:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: mon host = "[v1:172.21.15.67:6789],[v1:172.21.15.67:6790],[v1:172.21.15.82:6789]" 2024-03-29T11:18:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: lockdep = True 2024-03-29T11:18:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: mon client directed command retry = 5 2024-03-29T11:18:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: mon election default strategy = 1 2024-03-29T11:18:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: ms bind msgr2 = False 2024-03-29T11:18:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: ms inject socket failures = 5000 2024-03-29T11:18:48.828 INFO:teuthology.orchestra.run.smithi082.stdout: ms type = async 2024-03-29T11:18:48.828 INFO:teuthology.orchestra.run.smithi082.stdout: fsid = 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:18:48.828 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.828 INFO:teuthology.orchestra.run.smithi082.stdout:[osd] 2024-03-29T11:18:48.828 INFO:teuthology.orchestra.run.smithi082.stdout: osd journal size = 100 2024-03-29T11:18:48.828 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.828 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub load threshold = 5.0 2024-03-29T11:18:48.828 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub max interval = 600 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: osd recover clone overlap = true 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: osd recovery max chunk = 1048576 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug shutdown = true 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug op order = true 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify stray on activate = true 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug trim objects = true 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: osd open classes on start = true 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug pg log writeout = true 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:18:48.829 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.830 INFO:teuthology.orchestra.run.smithi082.stdout: osd map max advance = 10 2024-03-29T11:18:48.830 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.830 INFO:teuthology.orchestra.run.smithi082.stdout: journal zero on create = true 2024-03-29T11:18:48.830 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.830 INFO:teuthology.orchestra.run.smithi082.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:18:48.830 INFO:teuthology.orchestra.run.smithi082.stdout: filestore apply finisher threads = 3 2024-03-29T11:18:48.830 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.830 INFO:teuthology.orchestra.run.smithi082.stdout: bdev debug aio = true 2024-03-29T11:18:48.830 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug misdirected ops = true 2024-03-29T11:18:48.830 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore block size = 96636764160 2024-03-29T11:18:48.830 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore compression algorithm = snappy 2024-03-29T11:18:48.830 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore compression mode = aggressive 2024-03-29T11:18:48.830 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore fsck on mount = True 2024-03-29T11:18:48.831 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore zero block detection = True 2024-03-29T11:18:48.831 INFO:teuthology.orchestra.run.smithi082.stdout: debug bluefs = 20 2024-03-29T11:18:48.831 INFO:teuthology.orchestra.run.smithi082.stdout: debug bluestore = 20 2024-03-29T11:18:48.831 INFO:teuthology.orchestra.run.smithi082.stdout: debug monc = 20 2024-03-29T11:18:48.831 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-03-29T11:18:48.831 INFO:teuthology.orchestra.run.smithi082.stdout: debug osd = 20 2024-03-29T11:18:48.831 INFO:teuthology.orchestra.run.smithi082.stdout: debug rocksdb = 10 2024-03-29T11:18:48.831 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:48.831 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd full ratio = 0.9 2024-03-29T11:18:48.831 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:18:48.831 INFO:teuthology.orchestra.run.smithi082.stdout: osd class default list = * 2024-03-29T11:18:48.831 INFO:teuthology.orchestra.run.smithi082.stdout: osd class load list = * 2024-03-29T11:18:48.831 INFO:teuthology.orchestra.run.smithi082.stdout: osd client watch timeout = 120 2024-03-29T11:18:48.832 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify cached snaps = True 2024-03-29T11:18:48.832 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify missing on start = True 2024-03-29T11:18:48.832 INFO:teuthology.orchestra.run.smithi082.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:18:48.832 INFO:teuthology.orchestra.run.smithi082.stdout: osd mclock override recovery settings = True 2024-03-29T11:18:48.832 INFO:teuthology.orchestra.run.smithi082.stdout: osd objectstore = bluestore 2024-03-29T11:18:48.832 INFO:teuthology.orchestra.run.smithi082.stdout: osd op queue = debug_random 2024-03-29T11:18:48.832 INFO:teuthology.orchestra.run.smithi082.stdout: osd op queue cut off = debug_random 2024-03-29T11:18:48.832 INFO:teuthology.orchestra.run.smithi082.stdout: osd shutdown pgref assert = True 2024-03-29T11:18:48.832 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.832 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr] 2024-03-29T11:18:48.832 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-03-29T11:18:48.832 INFO:teuthology.orchestra.run.smithi082.stdout: debug mgr = 20 2024-03-29T11:18:48.833 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2024-03-29T11:18:48.833 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2024-03-29T11:18:48.833 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:48.833 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:48.833 INFO:teuthology.orchestra.run.smithi082.stdout: mgr/telemetry/nag = false 2024-03-29T11:18:48.833 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.833 INFO:teuthology.orchestra.run.smithi082.stdout:[mon] 2024-03-29T11:18:48.833 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-03-29T11:18:48.834 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2024-03-29T11:18:48.834 INFO:teuthology.orchestra.run.smithi082.stdout: debug paxos = 20 2024-03-29T11:18:48.834 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2024-03-29T11:18:48.834 INFO:teuthology.orchestra.run.smithi082.stdout: mon data avail warn = 5 2024-03-29T11:18:48.834 INFO:teuthology.orchestra.run.smithi082.stdout: mon mgr mkfs grace = 240 2024-03-29T11:18:48.834 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:48.834 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd reporter subtree level = osd 2024-03-29T11:18:48.834 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd prime pg temp = true 2024-03-29T11:18:48.835 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:48.835 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.835 INFO:teuthology.orchestra.run.smithi082.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:18:48.835 INFO:teuthology.orchestra.run.smithi082.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:18:48.835 INFO:teuthology.orchestra.run.smithi082.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:18:48.835 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.835 INFO:teuthology.orchestra.run.smithi082.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:18:48.835 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:18:48.836 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:18:48.836 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.836 INFO:teuthology.orchestra.run.smithi082.stdout: # 1m isn't quite enough 2024-03-29T11:18:48.836 INFO:teuthology.orchestra.run.smithi082.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:18:48.836 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.836 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:18:48.836 INFO:teuthology.orchestra.run.smithi082.stdout: mon min osdmap epochs = 50 2024-03-29T11:18:48.836 INFO:teuthology.orchestra.run.smithi082.stdout: mon osdmap full prune interval = 2 2024-03-29T11:18:48.837 INFO:teuthology.orchestra.run.smithi082.stdout: mon osdmap full prune min = 15 2024-03-29T11:18:48.837 INFO:teuthology.orchestra.run.smithi082.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:18:48.837 INFO:teuthology.orchestra.run.smithi082.stdout: mon scrub interval = 300 2024-03-29T11:18:48.837 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on pool no app = False 2024-03-29T11:18:48.837 INFO:teuthology.orchestra.run.smithi082.stdout: paxos service trim min = 10 2024-03-29T11:18:48.837 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T11:18:48.837 INFO:teuthology.orchestra.run.smithi082.stdout:[client] 2024-03-29T11:18:48.838 INFO:teuthology.orchestra.run.smithi082.stdout: rgw cache enabled = true 2024-03-29T11:18:48.838 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable ops log = true 2024-03-29T11:18:48.838 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable usage log = true 2024-03-29T11:18:48.838 INFO:teuthology.orchestra.run.smithi082.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:18:48.838 INFO:teuthology.orchestra.run.smithi082.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:18:48.838 INFO:teuthology.orchestra.run.smithi082.stdout: debug monc = 20 2024-03-29T11:18:48.839 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-03-29T11:18:48.839 INFO:teuthology.orchestra.run.smithi082.stdout: debug objecter = 20 2024-03-29T11:18:48.839 INFO:teuthology.orchestra.run.smithi082.stdout: debug rados = 20 2024-03-29T11:18:48.839 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.a] 2024-03-29T11:18:48.839 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.c] 2024-03-29T11:18:48.840 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.b] 2024-03-29T11:18:48.842 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:18:48.842 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:18:48.937 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:18:48.937 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:18:48.937 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:18:48.951 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:18:48.952 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:18:49.005 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-03-29T11:18:49.006 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:18:49.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:18:49.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:49.084 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:18:49.085 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:49.139 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T11:18:49.140 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:18:49.140 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:18:49.140 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:49.186 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:18:49.186 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:49.243 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:18:49.243 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:18:49.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:18:49.297 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:18:49.299 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:18:49.366 INFO:teuthology.orchestra.run.smithi082.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:18:49.369 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:18:49.369 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:18:49.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:18:49.413 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:18:49.426 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:18:49.470 INFO:teuthology.orchestra.run.smithi082.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:18:49.486 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:18:49.486 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:18:49.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:49.512 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:49.513 INFO:tasks.ceph:role: osd.0 2024-03-29T11:18:49.513 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-29T11:18:49.513 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:18:49.586 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:49.587 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:49.587 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:49.587 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:49.587 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:49.587 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:49.587 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:49.587 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:49.587 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:49.587 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:50.107 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-03-29T11:18:50.109 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-03-29T11:18:50.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:50.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:50.292 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:50.360 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:50.360 INFO:tasks.ceph:role: osd.1 2024-03-29T11:18:50.360 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-29T11:18:50.360 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:18:50.436 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:50.436 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:50.436 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:50.436 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:50.436 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:50.436 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:50.436 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:50.437 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:50.437 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:50.437 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:50.889 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-03-29T11:18:50.891 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-03-29T11:18:50.891 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:50.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:51.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:51.097 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:51.097 INFO:tasks.ceph:role: osd.2 2024-03-29T11:18:51.097 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-29T11:18:51.097 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:18:51.172 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:51.173 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:51.173 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:51.173 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:51.173 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:51.173 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:51.173 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:51.173 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:51.173 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:51.173 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:51.666 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-03-29T11:18:51.668 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-03-29T11:18:51.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:51.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:51.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:51.918 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:51.918 INFO:tasks.ceph:role: osd.3 2024-03-29T11:18:51.918 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-29T11:18:51.918 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:18:51.992 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:51.993 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:51.993 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:51.993 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:51.993 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:51.993 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:51.993 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:51.993 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:51.993 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:51.993 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:52.493 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-03-29T11:18:52.494 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-03-29T11:18:52.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:52.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:52.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:52.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:52.797+0000 7f599b391640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:18:52.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:52.798+0000 7f599b391640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:18:52.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:52.798+0000 7f599b391640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:52.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:52.798+0000 7f599b391640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:52.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:52.798+0000 7f599b391640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:52.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:52.798+0000 7f599b391640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:18:55.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:55.659 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:55.774+0000 7fd01a810640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:18:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:55.775+0000 7fd01a810640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:18:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:55.775+0000 7fd01a810640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:55.775+0000 7fd01a810640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:55.775+0000 7fd01a810640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:55.775+0000 7fd01a810640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:18:58.605 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:58.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:58.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:58.751+0000 7fd56971f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:18:58.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:58.752+0000 7fd56971f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:18:58.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:58.752+0000 7fd56971f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:58.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:58.752+0000 7fd56971f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:58.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:58.752+0000 7fd56971f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:58.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:18:58.752+0000 7fd56971f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:19:01.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:01.612 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:01.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:01.727+0000 7f75b5813640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:19:01.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:01.727+0000 7f75b5813640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:19:01.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:01.727+0000 7f75b5813640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:01.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:01.727+0000 7f75b5813640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:01.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:01.727+0000 7f75b5813640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:01.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:01.728+0000 7f75b5813640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:19:04.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:04.586 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:04.618 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:04.618 INFO:tasks.ceph:role: osd.4 2024-03-29T11:19:04.618 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T11:19:04.618 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:19:04.695 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:04.696 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:04.696 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:04.696 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:04.696 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:04.696 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:04.696 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:04.696 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:04.696 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:04.696 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:05.211 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-03-29T11:19:05.213 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-03-29T11:19:05.214 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:05.342 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:05.378 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:05.448 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:05.448 INFO:tasks.ceph:role: osd.5 2024-03-29T11:19:05.448 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T11:19:05.448 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:19:05.523 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:05.523 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:05.524 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:05.524 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:05.524 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:05.524 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:05.524 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:05.524 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:05.524 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:05.524 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:05.969 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-03-29T11:19:05.971 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-03-29T11:19:05.971 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:06.065 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:06.095 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:06.162 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:06.162 INFO:tasks.ceph:role: osd.6 2024-03-29T11:19:06.163 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T11:19:06.163 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:19:06.237 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:06.238 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:06.238 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:06.238 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:06.238 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:06.238 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:06.238 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:06.238 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:06.238 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:06.238 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:06.739 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-03-29T11:19:06.741 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-03-29T11:19:06.741 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:06.875 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:06.905 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:06.974 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:06.974 INFO:tasks.ceph:role: osd.7 2024-03-29T11:19:06.975 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T11:19:06.975 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:19:07.049 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:07.049 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:07.049 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:07.049 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:07.049 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:07.049 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:07.049 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:07.049 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:07.049 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:07.050 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:07.566 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-03-29T11:19:07.568 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-03-29T11:19:07.568 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:07.718 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:07.752 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:07.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:07.888+0000 7f7946e11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:19:07.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:07.888+0000 7f7946e11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:19:07.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:07.888+0000 7f7946e11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:07.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:07.888+0000 7f7946e11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:07.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:07.888+0000 7f7946e11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:07.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:07.888+0000 7f7946e11640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:19:10.714 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:10.745 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:10.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:10.861+0000 7f76748ae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:19:10.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:10.861+0000 7f76748ae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:19:10.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:10.861+0000 7f76748ae640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:10.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:10.861+0000 7f76748ae640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:10.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:10.861+0000 7f76748ae640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:10.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:10.862+0000 7f76748ae640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:19:13.680 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:13.711 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:13.802 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:13.825+0000 7f4ff5ff8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:19:13.802 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:13.825+0000 7f4ff5ff8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:19:13.802 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:13.825+0000 7f4ff5ff8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:13.803 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:13.825+0000 7f4ff5ff8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:13.803 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:13.825+0000 7f4ff5ff8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:13.803 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:13.826+0000 7f4ff5ff8640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:19:16.653 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:16.682 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:16.773 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:16.796+0000 7f3f461dd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:19:16.773 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:16.796+0000 7f3f461dd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:19:16.773 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:16.796+0000 7f3f461dd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:16.773 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:16.796+0000 7f3f461dd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:16.773 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:16.796+0000 7f3f461dd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:16.774 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:16.796+0000 7f3f461dd640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:19:19.628 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:19.660 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:19:19.660 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:19:19.660 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:19:19.692 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:19:19.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:19:19.762 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:19:19.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:19:19.831 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:19:19.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:19:19.900 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:19:19.900 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:19:19.969 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:19:19.969 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:19:20.000 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:19:20.001 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:19:20.071 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:19:20.071 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:19:20.141 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:19:20.142 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:19:20.211 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:19:20.211 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:19:20.280 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:19:20.280 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:19:20.297 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:19:20.297 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:19:20.336 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:19:20.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:19:20.339 DEBUG:teuthology.orchestra.run.smithi082:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:19:20.380 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.y] 2024-03-29T11:19:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCZowZmlG0qExAA0Hl4oCTOIexdldNBqSocPw== 2024-03-29T11:19:20.380 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-03-29T11:19:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCcowZmXjOYLxAAxGz9h7dfKxUPpyxc7KG7yQ== 2024-03-29T11:19:20.380 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-03-29T11:19:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCfowZmHyU6LhAAPEhP9iMgBydGInB3p5gQsw== 2024-03-29T11:19:20.380 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCiowZmBHzbLBAAeXridYrPOatsBwEzX14frw== 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQClowZmry9lKxAAvrmG6rdYCATgwchtK2Cdsw== 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCZowZmVudDFxAAuyikR2bpA/LmzmeTx1x8dA== 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCrowZmEqn/NBAA587qinLrhFpkyE263KXfOg== 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCuowZmRWZmMxAA+7L7uSoYXlyZIZRcM3TwwQ== 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.6] 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCxowZmyBRBMRAAFLPz5Q3mFOBxtSNcfB3XpA== 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.7] 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQC0owZmHg2ELxAADimovhieUdM7yn5x8eN7fw== 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-03-29T11:19:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCZowZm67L3GRAAMXR1A/ST/r4zCBElAkKSfw== 2024-03-29T11:19:20.382 INFO:teuthology.orchestra.run.smithi067.stdout:[client.1] 2024-03-29T11:19:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCZowZmhO2CHRAA7A4MDTuc4clMAl2MejFfTg== 2024-03-29T11:19:20.403 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr.y] 2024-03-29T11:19:20.403 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCZowZmlG0qExAA0Hl4oCTOIexdldNBqSocPw== 2024-03-29T11:19:20.403 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.0] 2024-03-29T11:19:20.403 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCcowZmXjOYLxAAxGz9h7dfKxUPpyxc7KG7yQ== 2024-03-29T11:19:20.403 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.1] 2024-03-29T11:19:20.403 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCfowZmHyU6LhAAPEhP9iMgBydGInB3p5gQsw== 2024-03-29T11:19:20.403 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.2] 2024-03-29T11:19:20.403 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCiowZmBHzbLBAAeXridYrPOatsBwEzX14frw== 2024-03-29T11:19:20.403 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.3] 2024-03-29T11:19:20.403 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQClowZmry9lKxAAvrmG6rdYCATgwchtK2Cdsw== 2024-03-29T11:19:20.403 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr.x] 2024-03-29T11:19:20.404 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCZowZmVudDFxAAuyikR2bpA/LmzmeTx1x8dA== 2024-03-29T11:19:20.404 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.4] 2024-03-29T11:19:20.404 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCrowZmEqn/NBAA587qinLrhFpkyE263KXfOg== 2024-03-29T11:19:20.404 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.5] 2024-03-29T11:19:20.404 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCuowZmRWZmMxAA+7L7uSoYXlyZIZRcM3TwwQ== 2024-03-29T11:19:20.404 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.6] 2024-03-29T11:19:20.404 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCxowZmyBRBMRAAFLPz5Q3mFOBxtSNcfB3XpA== 2024-03-29T11:19:20.404 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.7] 2024-03-29T11:19:20.404 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQC0owZmHg2ELxAADimovhieUdM7yn5x8eN7fw== 2024-03-29T11:19:20.404 INFO:teuthology.orchestra.run.smithi082.stdout:[client.0] 2024-03-29T11:19:20.404 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCZowZm67L3GRAAMXR1A/ST/r4zCBElAkKSfw== 2024-03-29T11:19:20.404 INFO:teuthology.orchestra.run.smithi082.stdout:[client.1] 2024-03-29T11:19:20.404 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCZowZmhO2CHRAA7A4MDTuc4clMAl2MejFfTg== 2024-03-29T11:19:20.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:20.425 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:20.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.501 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.555 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.609 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.663 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:20.717 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:20.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.770 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.822 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.824 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.875 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.878 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.931 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:20.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:20.985 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:21.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:21.042 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:21.096 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:19:21.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:19:21.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:21.219 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:19:21.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:19:21.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:21.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:19:21.436 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:19:21.464 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:21.562 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:19:21.588 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:21.591 DEBUG:teuthology.orchestra.run.smithi082:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:21.643 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:19:21.643 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:19:21.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:19:21.646 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:19:21.646 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:19:21.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:19:21.648 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:19:21.648 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:19:21.648 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:19:21.686 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:19:21.686 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:19:21.686 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:19:21.686 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:19:21.688 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:19:21.689 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:19:21.689 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:19:21.690 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:19:21.691 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:19:21.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:19:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.817+0000 7f07ab9b5640 10 monclient: get_monmap_and_config 2024-03-29T11:19:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.817+0000 7f07ab9b5640 10 monclient: build_initial_monmap 2024-03-29T11:19:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.818+0000 7f07ab9b5640 10 monclient: monmap: 2024-03-29T11:19:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-03-29T11:19:21.795 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:21.795 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-03-29T11:19:21.818771+0000 2024-03-29T11:19:21.795 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-03-29T11:19:21.818771+0000 2024-03-29T11:19:21.795 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:21.795 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:21.795 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:21.795 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:21.795 INFO:teuthology.orchestra.run.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:21.795 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:19:21.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.818+0000 7f07ab9b5640 1 Processor -- start 2024-03-29T11:19:21.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.818+0000 7f07ab9b5640 1 -- start start 2024-03-29T11:19:21.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.818+0000 7f07ab9b5640 10 monclient: init 2024-03-29T11:19:21.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.819+0000 7f07ab9b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:21.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.819+0000 7f07ab9b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:21.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.819+0000 7f07ab9b5640 10 monclient(hunting): picked mon.noname-a con 0x7f07a40b8240 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:21.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.819+0000 7f07ab9b5640 10 monclient(hunting): picked mon.noname-b con 0x7f07a40b1f60 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:21.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.819+0000 7f07ab9b5640 10 monclient(hunting): picked mon.noname-c con 0x7f07a409c8c0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:21.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.819+0000 7f07ab9b5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a409bd80 con 0x7f07a40b8240 2024-03-29T11:19:21.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.819+0000 7f07ab9b5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a409b960 con 0x7f07a409c8c0 2024-03-29T11:19:21.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.819+0000 7f07ab9b5640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40b2ec0 con 0x7f07a40b1f60 2024-03-29T11:19:21.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.819+0000 7f07ab9b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:21.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.819+0000 7f07ab9b5640 10 monclient(hunting): authenticate will time out at 822.115261s 2024-03-29T11:19:21.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.819+0000 7f07a8f50640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f07a40b1f60 0x7f07a409c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:35438/0 (socket says 172.21.15.67:35438) 2024-03-29T11:19:21.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:21.819+0000 7f07a8f50640 1 -- 172.21.15.67:0/2827752071 learned_addr learned my addr 172.21.15.67:0/2827752071 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:19:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 10 monclient(hunting): tick 2024-03-29T11:19:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:19:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 1 monclient(hunting): continuing hunt 2024-03-29T11:19:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:19:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 1 -- 172.21.15.67:0/2827752071 >> v1:172.21.15.67:6790/0 conn(0x7f07a40b1f60 legacy=0x7f07a409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 1 -- 172.21.15.67:0/2827752071 >> v1:172.21.15.82:6789/0 conn(0x7f07a409c8c0 legacy=0x7f07a414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 1 -- 172.21.15.67:0/2827752071 >> v1:172.21.15.67:6789/0 conn(0x7f07a40b8240 legacy=0x7f07a413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 10 monclient(hunting): picked mon.noname-a con 0x7f07880036e0 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 10 monclient(hunting): picked mon.noname-c con 0x7f07a409c8c0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 10 monclient(hunting): picked mon.noname-b con 0x7f078800aa90 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 1 -- 172.21.15.67:0/2827752071 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a413fb60 con 0x7f07880036e0 2024-03-29T11:19:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 1 -- 172.21.15.67:0/2827752071 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f078800f760 con 0x7f07a409c8c0 2024-03-29T11:19:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 1 -- 172.21.15.67:0/2827752071 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07880109c0 con 0x7f078800aa90 2024-03-29T11:19:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:24.819+0000 7f079affd640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 1 -- 172.21.15.67:0/2827752071 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861634667 0 0) 0x7f07a413fb60 con 0x7f07880036e0 2024-03-29T11:19:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:19:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 1 -- 172.21.15.67:0/2827752071 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0790003520 con 0x7f07880036e0 2024-03-29T11:19:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 1 -- 172.21.15.67:0/2827752071 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582890413 0 0) 0x7f078800f760 con 0x7f07a409c8c0 2024-03-29T11:19:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:19:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 1 -- 172.21.15.67:0/2827752071 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a413fb60 con 0x7f07a409c8c0 2024-03-29T11:19:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 1 -- 172.21.15.67:0/2827752071 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067175819 0 0) 0x7f07880109c0 con 0x7f078800aa90 2024-03-29T11:19:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:19:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.988+0000 7f079bfff640 1 -- 172.21.15.67:0/2827752071 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f078800f760 con 0x7f078800aa90 2024-03-29T11:19:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 1 -- 172.21.15.67:0/2827752071 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1116111071 0 0) 0x7f07a413fb60 con 0x7f07a409c8c0 2024-03-29T11:19:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 1 -- 172.21.15.67:0/2827752071 >> v1:172.21.15.67:6790/0 conn(0x7f078800aa90 legacy=0x7f078800cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 1 -- 172.21.15.67:0/2827752071 >> v1:172.21.15.67:6789/0 conn(0x7f07880036e0 legacy=0x7f0788005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 1 monclient: found mon.noname-c 2024-03-29T11:19:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:19:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 1 -- 172.21.15.67:0/2827752071 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0788011c20 con 0x7f07a409c8c0 2024-03-29T11:19:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f07ab9b5640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:19:26.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f07ab9b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:26.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 1 -- 172.21.15.67:0/2827752071 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f07a0001990 con 0x7f07a409c8c0 2024-03-29T11:19:26.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:26.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 10 monclient: dump: 2024-03-29T11:19:26.970 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-03-29T11:19:26.970 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:26.970 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:26.970 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:26.970 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.970 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:26.970 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:26.970 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f07ab9b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 1 -- 172.21.15.67:0/2827752071 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07a0001af0 con 0x7f07a409c8c0 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f07ab9b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f07ab9b5640 4 set_mon_vals no callback set 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 1 -- 172.21.15.67:0/2827752071 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f07a0001f00 con 0x7f07a409c8c0 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.989+0000 7f079bfff640 10 monclient: dump: 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:26.971 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.972 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:26.972 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:26.972 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:26.972 INFO:teuthology.orchestra.run.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:26.972 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:19:26.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.990+0000 7f07ab9b5640 10 monclient: shutdown 2024-03-29T11:19:26.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.990+0000 7f07ab9b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.990+0000 7f07ab9b5640 1 -- 172.21.15.67:0/2827752071 >> v1:172.21.15.82:6789/0 conn(0x7f07a409c8c0 legacy=0x7f0788009320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.990+0000 7f07ab9b5640 1 -- 172.21.15.67:0/2827752071 shutdown_connections 2024-03-29T11:19:26.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.990+0000 7f07ab9b5640 1 -- 172.21.15.67:0/2827752071 >> 172.21.15.67:0/2827752071 conn(0x7f07a4066b00 msgr2=0x7f07a4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.990+0000 7f07ab9b5640 1 -- 172.21.15.67:0/2827752071 shutdown_connections 2024-03-29T11:19:26.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.990+0000 7f07ab9b5640 1 -- 172.21.15.67:0/2827752071 wait complete. 2024-03-29T11:19:26.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.991+0000 7f07ab9b5640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.991+0000 7f07ab9b5640 10 monclient: monmap: 2024-03-29T11:19:26.978 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-03-29T11:19:26.978 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:26.978 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2024-03-29T11:19:26.978 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2024-03-29T11:19:26.978 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.978 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:26.978 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:26.978 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:26.978 INFO:teuthology.orchestra.run.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:26.978 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:19:26.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.992+0000 7f07ab9b5640 1 librados: starting msgr at 2024-03-29T11:19:26.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.992+0000 7f07ab9b5640 1 librados: starting objecter 2024-03-29T11:19:26.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.992+0000 7f07ab9b5640 1 Processor -- start 2024-03-29T11:19:26.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.992+0000 7f07ab9b5640 1 -- start start 2024-03-29T11:19:26.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.992+0000 7f07ab9b5640 1 librados: setting wanted keys 2024-03-29T11:19:26.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.992+0000 7f07ab9b5640 1 librados: calling monclient init 2024-03-29T11:19:26.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.992+0000 7f07ab9b5640 10 monclient: init 2024-03-29T11:19:26.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f07ab9b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f07ab9b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:26.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f07ab9b5640 10 monclient(hunting): picked mon.noname-c con 0x7f07880036e0 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:26.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f07ab9b5640 10 monclient(hunting): picked mon.noname-b con 0x7f078800aa90 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:26.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f07ab9b5640 10 monclient(hunting): picked mon.noname-a con 0x7f07a409c8c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:26.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f07ab9b5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a41d4a20 con 0x7f07a409c8c0 2024-03-29T11:19:26.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f07ab9b5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40b1df0 con 0x7f078800aa90 2024-03-29T11:19:26.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f07a9751640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f07880036e0 0x7f07a40ae0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:35594/0 (socket says 172.21.15.67:35594) 2024-03-29T11:19:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f07a9751640 1 -- 172.21.15.67:0/3545033577 learned_addr learned my addr 172.21.15.67:0/3545033577 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:19:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f07ab9b5640 1 -- 172.21.15.67:0/3545033577 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40b0900 con 0x7f07880036e0 2024-03-29T11:19:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f0799ffb640 10 client.?.objecter ms_handle_connect 0x7f07880036e0 2024-03-29T11:19:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f0799ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f0799ffb640 10 client.?.objecter ms_handle_connect 0x7f078800aa90 2024-03-29T11:19:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f0799ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f0799ffb640 10 client.?.objecter ms_handle_connect 0x7f07a409c8c0 2024-03-29T11:19:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f0799ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f07ab9b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f07ab9b5640 10 monclient(hunting): authenticate will time out at 827.289684s 2024-03-29T11:19:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097077952 0 0) 0x7f07a41d4a20 con 0x7f07a409c8c0 2024-03-29T11:19:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f0799ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f0799ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f0799ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f0799ffb640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:19:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.993+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07880100b0 con 0x7f07a409c8c0 2024-03-29T11:19:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1566954443 0 0) 0x7f07880100b0 con 0x7f07a409c8c0 2024-03-29T11:19:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07a41d4a20 con 0x7f07a409c8c0 2024-03-29T11:19:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f079c001850 con 0x7f07a409c8c0 2024-03-29T11:19:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-03-29T11:19:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:26.982 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.982 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:26.982 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:26.982 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:26.982 INFO:teuthology.orchestra.run.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:26.982 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:19:26.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:26.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1044001570 0 0) 0x7f07a41d4a20 con 0x7f07a409c8c0 2024-03-29T11:19:26.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 >> v1:172.21.15.67:6790/0 conn(0x7f07880036e0 legacy=0x7f07a40ae0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 >> v1:172.21.15.82:6789/0 conn(0x7f078800aa90 legacy=0x7f07a40ae7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 1 monclient: found mon.a 2024-03-29T11:19:26.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:26.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a40b0a90 con 0x7f07a409c8c0 2024-03-29T11:19:26.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f07ab9b5640 5 monclient: authenticate success, global_id 4113 2024-03-29T11:19:26.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f07ab9b5640 10 monclient: _renew_subs 2024-03-29T11:19:26.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f07ab9b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:26.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f07ab9b5640 1 -- 172.21.15.67:0/3545033577 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a40b14e0 con 0x7f07a409c8c0 2024-03-29T11:19:26.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f079c0035f0 con 0x7f07a409c8c0 2024-03-29T11:19:26.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f079c003900 con 0x7f07a409c8c0 2024-03-29T11:19:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 10 monclient: dump: 2024-03-29T11:19:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-03-29T11:19:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:26.984 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:19:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f079c003df0 con 0x7f07a409c8c0 2024-03-29T11:19:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.995+0000 7f07ab9b5640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.995+0000 7f07ab9b5640 10 monclient: _renew_subs 2024-03-29T11:19:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.995+0000 7f07ab9b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.994+0000 7f079b7fe640 4 set_mon_vals no callback set 2024-03-29T11:19:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.995+0000 7f07ab9b5640 1 -- 172.21.15.67:0/3545033577 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a41e3190 con 0x7f07a409c8c0 2024-03-29T11:19:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.995+0000 7f07ab9b5640 1 librados: init done 2024-03-29T11:19:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.996+0000 7f07ab9b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.996+0000 7f07ab9b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.996+0000 7f07ab9b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.996+0000 7f07ab9b5640 1 -- 172.21.15.67:0/3545033577 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a409b170 con 0x7f07a409c8c0 2024-03-29T11:19:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.997+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3026403013 0 0) 0x7f079c0043e0 con 0x7f07a409c8c0 2024-03-29T11:19:26.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.997+0000 7f0799ffb640 10 client.4113.objecter ms_dispatch 0x7f07a4000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:19:26.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.997+0000 7f0799ffb640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:19:26.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.997+0000 7f0799ffb640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:19:26.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.997+0000 7f0799ffb640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T11:19:26.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.999+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f079c004710 con 0x7f07a409c8c0 2024-03-29T11:19:26.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.999+0000 7f0799ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:26.999+0000 7f0799ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:27.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.069+0000 7f07ab9b5640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:27.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.069+0000 7f07ab9b5640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:27.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.069+0000 7f07ab9b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:27.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.069+0000 7f07ab9b5640 1 -- 172.21.15.67:0/3545033577 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f07a40b0c80 con 0x7f07a409c8c0 2024-03-29T11:19:27.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.071+0000 7f0799ffb640 1 -- 172.21.15.67:0/3545033577 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f079c004970 con 0x7f07a409c8c0 2024-03-29T11:19:27.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.071+0000 7f0799ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:27.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.071+0000 7f0799ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:19:27.048 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-03-29T11:19:27.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.072+0000 7f07ab9b5640 10 librados: watch_flush enter 2024-03-29T11:19:27.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.072+0000 7f07ab9b5640 10 librados: watch_flush exit 2024-03-29T11:19:27.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.072+0000 7f07ab9b5640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T11:19:27.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.072+0000 7f07ab9b5640 10 client.4113.objecter successfully canceled tick 2024-03-29T11:19:27.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.072+0000 7f07ab9b5640 10 monclient: shutdown 2024-03-29T11:19:27.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.072+0000 7f07ab9b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:27.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.072+0000 7f07ab9b5640 1 -- 172.21.15.67:0/3545033577 >> v1:172.21.15.67:6789/0 conn(0x7f07a409c8c0 legacy=0x7f07a41e1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.072+0000 7f07ab9b5640 1 -- 172.21.15.67:0/3545033577 shutdown_connections 2024-03-29T11:19:27.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.072+0000 7f07ab9b5640 1 -- 172.21.15.67:0/3545033577 >> 172.21.15.67:0/3545033577 conn(0x7f07a4066b00 msgr2=0x7f07a40b5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:27.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.072+0000 7f07ab9b5640 1 -- 172.21.15.67:0/3545033577 shutdown_connections 2024-03-29T11:19:27.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.072+0000 7f07ab9b5640 1 -- 172.21.15.67:0/3545033577 wait complete. 2024-03-29T11:19:27.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:27.072+0000 7f07ab9b5640 1 librados: shutdown 2024-03-29T11:19:27.061 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:19:27.061 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:19:27.061 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:19:27.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:19:27.089 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:19:27.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:19:27.155 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:19:27.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:19:27.222 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-29T11:19:27.222 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:19:27.289 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:19:27.289 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:19:27.316 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:19:27.316 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:19:27.383 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:19:27.383 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:19:27.430 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:27.453+0000 7fcb6aa38040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:27.450 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T11:19:27.451 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:19:27.479 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:27.502+0000 7f0c9364b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:27.486 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:27.510+0000 7fcb6aa38040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:27.513 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 90c40155-32e1-4ef7-a4c5-1f1de26a6f4c 0 2024-03-29T11:19:27.535 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:27.558+0000 7f0c9364b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:27.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.676+0000 7feb4befb640 10 monclient: get_monmap_and_config 2024-03-29T11:19:27.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.676+0000 7feb4befb640 10 monclient: build_initial_monmap 2024-03-29T11:19:27.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.676+0000 7feb4befb640 10 monclient: monmap: 2024-03-29T11:19:27.654 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:27.654 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:27.654 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:19:27.677650+0000 2024-03-29T11:19:27.654 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:19:27.677650+0000 2024-03-29T11:19:27.654 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:27.654 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.654 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:27.654 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:27.654 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:27.654 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.677+0000 7feb4befb640 1 Processor -- start 2024-03-29T11:19:27.664 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.677+0000 7feb4befb640 1 -- start start 2024-03-29T11:19:27.664 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.677+0000 7feb4befb640 10 monclient: init 2024-03-29T11:19:27.664 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.677+0000 7feb4befb640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:27.664 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.677+0000 7feb4befb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:27.664 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.677+0000 7feb4befb640 10 monclient(hunting): picked mon.noname-c con 0x7feb440ae3c0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:27.664 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.677+0000 7feb4befb640 10 monclient(hunting): picked mon.noname-a con 0x7feb440aae90 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:27.664 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.677+0000 7feb4befb640 10 monclient(hunting): picked mon.noname-b con 0x7feb44071dc0 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:27.665 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.677+0000 7feb4befb640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb440ac7f0 con 0x7feb440aae90 2024-03-29T11:19:27.665 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.677+0000 7feb4befb640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb440ab370 con 0x7feb440ae3c0 2024-03-29T11:19:27.665 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.677+0000 7feb4befb640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb440ab890 con 0x7feb44071dc0 2024-03-29T11:19:27.665 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.677+0000 7feb4befb640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:27.665 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.677+0000 7feb4befb640 10 monclient(hunting): authenticate will time out at 822.949132s 2024-03-29T11:19:27.665 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb49c97640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7feb440ae3c0 0x7feb4413c290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37720/0 (socket says 172.21.15.82:37720) 2024-03-29T11:19:27.665 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb49c97640 1 -- 172.21.15.82:0/3984649055 learned_addr learned my addr 172.21.15.82:0/3984649055 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:27.665 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 1 -- 172.21.15.82:0/3984649055 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585792170 0 0) 0x7feb440ab370 con 0x7feb440ae3c0 2024-03-29T11:19:27.665 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.665 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.665 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.666 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:19:27.666 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 1 -- 172.21.15.82:0/3984649055 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb30003520 con 0x7feb440ae3c0 2024-03-29T11:19:27.666 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 1 -- 172.21.15.82:0/3984649055 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105370502 0 0) 0x7feb440ac7f0 con 0x7feb440aae90 2024-03-29T11:19:27.666 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.666 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.666 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.666 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:19:27.666 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 1 -- 172.21.15.82:0/3984649055 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb440ab370 con 0x7feb440aae90 2024-03-29T11:19:27.666 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 1 -- 172.21.15.82:0/3984649055 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975233730 0 0) 0x7feb440ab890 con 0x7feb44071dc0 2024-03-29T11:19:27.666 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.667 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.667 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.667 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:19:27.667 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 1 -- 172.21.15.82:0/3984649055 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb440ac7f0 con 0x7feb44071dc0 2024-03-29T11:19:27.667 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 1 -- 172.21.15.82:0/3984649055 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2869772020 0 0) 0x7feb30003520 con 0x7feb440ae3c0 2024-03-29T11:19:27.667 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 1 -- 172.21.15.82:0/3984649055 >> v1:172.21.15.67:6790/0 conn(0x7feb44071dc0 legacy=0x7feb4414a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.667 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 1 -- 172.21.15.82:0/3984649055 >> v1:172.21.15.67:6789/0 conn(0x7feb440aae90 legacy=0x7feb440acef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.667 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient: _finish_hunting 0 2024-03-29T11:19:27.667 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 1 monclient: found mon.noname-c 2024-03-29T11:19:27.667 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:27.667 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:19:27.667 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 1 -- 172.21.15.82:0/3984649055 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb440ad600 con 0x7feb440ae3c0 2024-03-29T11:19:27.668 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient: _finish_auth 0 2024-03-29T11:19:27.668 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient: _check_auth_tickets 2024-03-29T11:19:27.668 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:27.668 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 1 -- 172.21.15.82:0/3984649055 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7feb40001690 con 0x7feb440ae3c0 2024-03-29T11:19:27.668 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb4befb640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:19:27.668 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb4befb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:27.668 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.668 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:27.668 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient: dump: 2024-03-29T11:19:27.668 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:27.668 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:27.669 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.669 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.669 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.669 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.669 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:27.669 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:27.669 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:27.669 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.669 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb4befb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:27.669 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 1 -- 172.21.15.82:0/3984649055 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb40001870 con 0x7feb440ae3c0 2024-03-29T11:19:27.669 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:27.669 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 1 -- 172.21.15.82:0/3984649055 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7feb40001c80 con 0x7feb440ae3c0 2024-03-29T11:19:27.670 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb4befb640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:27.670 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb4befb640 4 set_mon_vals no callback set 2024-03-29T11:19:27.670 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.670 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:27.670 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.678+0000 7feb48c95640 10 monclient: dump: 2024-03-29T11:19:27.670 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:27.670 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:27.670 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.670 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.670 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.670 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.670 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:27.671 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:27.671 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:27.671 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.679+0000 7feb4befb640 10 monclient: shutdown 2024-03-29T11:19:27.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.679+0000 7feb4befb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:27.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.679+0000 7feb4befb640 1 -- 172.21.15.82:0/3984649055 >> v1:172.21.15.82:6789/0 conn(0x7feb440ae3c0 legacy=0x7feb4413c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.679+0000 7feb4befb640 1 -- 172.21.15.82:0/3984649055 shutdown_connections 2024-03-29T11:19:27.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.679+0000 7feb4befb640 1 -- 172.21.15.82:0/3984649055 >> 172.21.15.82:0/3984649055 conn(0x7feb440b3750 msgr2=0x7feb440b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:27.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.679+0000 7feb4befb640 1 -- 172.21.15.82:0/3984649055 shutdown_connections 2024-03-29T11:19:27.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.679+0000 7feb4befb640 1 -- 172.21.15.82:0/3984649055 wait complete. 2024-03-29T11:19:27.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.679+0000 7feb4befb640 10 monclient: build_initial_monmap 2024-03-29T11:19:27.671 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.679+0000 7feb4befb640 10 monclient: monmap: 2024-03-29T11:19:27.672 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:27.672 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:27.672 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T11:19:27.672 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T11:19:27.672 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:27.672 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.672 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:27.672 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:27.672 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:27.672 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.672 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.679+0000 7feb4befb640 1 librados: starting msgr at 2024-03-29T11:19:27.672 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.679+0000 7feb4befb640 1 librados: starting objecter 2024-03-29T11:19:27.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 1 Processor -- start 2024-03-29T11:19:27.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 1 -- start start 2024-03-29T11:19:27.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 1 librados: setting wanted keys 2024-03-29T11:19:27.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 1 librados: calling monclient init 2024-03-29T11:19:27.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 10 monclient: init 2024-03-29T11:19:27.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:27.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:27.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 10 monclient(hunting): picked mon.noname-b con 0x7feb44071dc0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:27.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 10 monclient(hunting): picked mon.noname-c con 0x7feb440aae90 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:27.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 10 monclient(hunting): picked mon.noname-a con 0x7feb440ae3c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:27.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb44132b30 con 0x7feb440ae3c0 2024-03-29T11:19:27.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4406e470 con 0x7feb44071dc0 2024-03-29T11:19:27.673 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4406e750 con 0x7feb440aae90 2024-03-29T11:19:27.674 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:27.674 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb4befb640 10 monclient(hunting): authenticate will time out at 822.951938s 2024-03-29T11:19:27.674 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb49c97640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7feb44071dc0 0x7feb4406ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37722/0 (socket says 172.21.15.82:37722) 2024-03-29T11:19:27.674 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb49c97640 1 -- 172.21.15.82:0/2035222909 learned_addr learned my addr 172.21.15.82:0/2035222909 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:27.674 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb3a7fc640 10 client.?.objecter ms_handle_connect 0x7feb44071dc0 2024-03-29T11:19:27.674 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:27.674 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb3a7fc640 10 client.?.objecter ms_handle_connect 0x7feb440aae90 2024-03-29T11:19:27.674 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:27.674 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb3a7fc640 10 client.?.objecter ms_handle_connect 0x7feb440ae3c0 2024-03-29T11:19:27.674 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:27.674 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941762548 0 0) 0x7feb4406e470 con 0x7feb44071dc0 2024-03-29T11:19:27.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb3a7fc640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:19:27.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.680+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb28004eb0 con 0x7feb44071dc0 2024-03-29T11:19:27.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026069284 0 0) 0x7feb4406e750 con 0x7feb440aae90 2024-03-29T11:19:27.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:19:27.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb4406e470 con 0x7feb440aae90 2024-03-29T11:19:27.675 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561139613 0 0) 0x7feb44132b30 con 0x7feb440ae3c0 2024-03-29T11:19:27.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:19:27.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb4406e750 con 0x7feb440ae3c0 2024-03-29T11:19:27.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1316684705 0 0) 0x7feb28004eb0 con 0x7feb44071dc0 2024-03-29T11:19:27.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb44132b30 con 0x7feb44071dc0 2024-03-29T11:19:27.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7feb40001c80 con 0x7feb44071dc0 2024-03-29T11:19:27.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:27.676 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:27.677 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:27.677 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:27.677 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.677 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.677 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.677 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.677 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:27.677 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:27.677 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:27.677 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:27.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 553314434 0 0) 0x7feb4406e750 con 0x7feb440ae3c0 2024-03-29T11:19:27.677 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb28004eb0 con 0x7feb440ae3c0 2024-03-29T11:19:27.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3586594702 0 0) 0x7feb4406e470 con 0x7feb440aae90 2024-03-29T11:19:27.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb4406e750 con 0x7feb440aae90 2024-03-29T11:19:27.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7feb3c001c70 con 0x7feb440ae3c0 2024-03-29T11:19:27.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:27.678 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:27.678 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:27.678 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:27.678 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.678 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.678 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.679 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.679 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:27.679 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:27.679 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:27.679 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1252151370 0 0) 0x7feb44132b30 con 0x7feb44071dc0 2024-03-29T11:19:27.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 >> v1:172.21.15.67:6790/0 conn(0x7feb440aae90 legacy=0x7feb4412f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 >> v1:172.21.15.67:6789/0 conn(0x7feb440ae3c0 legacy=0x7feb4412fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:27.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 monclient: found mon.b 2024-03-29T11:19:27.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:27.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:27.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4406ea30 con 0x7feb44071dc0 2024-03-29T11:19:27.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:27.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:27.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:27.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb4befb640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:19:27.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb4befb640 10 monclient: _renew_subs 2024-03-29T11:19:27.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb4befb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:27.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb4befb640 1 -- 172.21.15.82:0/2035222909 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb44132df0 con 0x7feb44071dc0 2024-03-29T11:19:27.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb4befb640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:27.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb4befb640 10 monclient: _renew_subs 2024-03-29T11:19:27.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb4befb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:27.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb4befb640 1 -- 172.21.15.82:0/2035222909 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb44132270 con 0x7feb44071dc0 2024-03-29T11:19:27.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb4befb640 1 librados: init done 2024-03-29T11:19:27.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb40001230 con 0x7feb44071dc0 2024-03-29T11:19:27.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:27.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7feb40004c30 con 0x7feb44071dc0 2024-03-29T11:19:27.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:27.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3bfff640 4 set_mon_vals no callback set 2024-03-29T11:19:27.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 monclient: dump: 2024-03-29T11:19:27.681 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:27.681 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:27.681 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.681 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.682 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.682 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.682 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:27.682 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:27.682 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:27.682 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7feb400051a0 con 0x7feb44071dc0 2024-03-29T11:19:27.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3586619731 0 0) 0x7feb40006760 con 0x7feb44071dc0 2024-03-29T11:19:27.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 10 client.4111.objecter ms_dispatch 0x7feb44000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:19:27.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:19:27.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:19:27.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.681+0000 7feb3a7fc640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T11:19:27.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.682+0000 7feb4befb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:27.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.682+0000 7feb4befb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:27.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.682+0000 7feb4befb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:27.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.682+0000 7feb4befb640 1 -- 172.21.15.82:0/2035222909 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2c005080 con 0x7feb44071dc0 2024-03-29T11:19:27.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.684+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7feb40006a40 con 0x7feb44071dc0 2024-03-29T11:19:27.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.684+0000 7feb3a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:27.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.684+0000 7feb3a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:27.733 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.756+0000 7feb4befb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "90c40155-32e1-4ef7-a4c5-1f1de26a6f4c", "id": 0}] 2024-03-29T11:19:27.733 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.756+0000 7feb4befb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "90c40155-32e1-4ef7-a4c5-1f1de26a6f4c", "id": 0}] 2024-03-29T11:19:27.733 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.756+0000 7feb4befb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:27.733 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.756+0000 7feb4befb640 1 -- 172.21.15.82:0/2035222909 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd new", "uuid": "90c40155-32e1-4ef7-a4c5-1f1de26a6f4c", "id": 0} v 0) v1 -- 0x7feb2c0058b0 con 0x7feb44071dc0 2024-03-29T11:19:27.738 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.761+0000 7feb3a7fc640 1 -- 172.21.15.82:0/2035222909 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "90c40155-32e1-4ef7-a4c5-1f1de26a6f4c", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 372774956 0 732707515) 0x7feb40005800 con 0x7feb44071dc0 2024-03-29T11:19:27.738 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.761+0000 7feb3a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "90c40155-32e1-4ef7-a4c5-1f1de26a6f4c", "id": 0}] 2024-03-29T11:19:27.738 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.761+0000 7feb3a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:27.739 INFO:teuthology.orchestra.run.smithi082.stdout:0 2024-03-29T11:19:27.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.762+0000 7feb4befb640 10 librados: watch_flush enter 2024-03-29T11:19:27.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.762+0000 7feb4befb640 10 librados: watch_flush exit 2024-03-29T11:19:27.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.762+0000 7feb4befb640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T11:19:27.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.762+0000 7feb4befb640 10 client.4111.objecter successfully canceled tick 2024-03-29T11:19:27.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.762+0000 7feb4befb640 10 monclient: shutdown 2024-03-29T11:19:27.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.762+0000 7feb4befb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:27.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.762+0000 7feb4befb640 1 -- 172.21.15.82:0/2035222909 >> v1:172.21.15.82:6789/0 conn(0x7feb44071dc0 legacy=0x7feb4406ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.762+0000 7feb4befb640 1 -- 172.21.15.82:0/2035222909 shutdown_connections 2024-03-29T11:19:27.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.762+0000 7feb4befb640 1 -- 172.21.15.82:0/2035222909 >> 172.21.15.82:0/2035222909 conn(0x7feb440b3750 msgr2=0x7feb44135e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:27.741 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.762+0000 7feb4befb640 1 -- 172.21.15.82:0/2035222909 shutdown_connections 2024-03-29T11:19:27.741 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.762+0000 7feb4befb640 1 -- 172.21.15.82:0/2035222909 wait complete. 2024-03-29T11:19:27.741 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.763+0000 7feb4befb640 1 librados: shutdown 2024-03-29T11:19:27.750 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 0269e563-f168-4fd0-9bf0-b38c6a7fb937 1 2024-03-29T11:19:27.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.871+0000 7f6fe43e6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:27.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.871+0000 7f6fe43e6640 10 monclient: build_initial_monmap 2024-03-29T11:19:27.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.871+0000 7f6fe43e6640 10 monclient: monmap: 2024-03-29T11:19:27.848 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:27.848 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:27.848 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:19:27.872274+0000 2024-03-29T11:19:27.848 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:19:27.872274+0000 2024-03-29T11:19:27.849 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:27.849 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.849 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:27.849 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:27.849 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:27.849 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.849 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.871+0000 7f6fe43e6640 1 Processor -- start 2024-03-29T11:19:27.849 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.871+0000 7f6fe43e6640 1 -- start start 2024-03-29T11:19:27.849 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.871+0000 7f6fe43e6640 10 monclient: init 2024-03-29T11:19:27.857 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe43e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:27.857 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe43e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:27.857 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe43e6640 10 monclient(hunting): picked mon.noname-b con 0x7f6fdc09ac40 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:27.857 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe43e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6fdc09c870 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:27.857 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe43e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6fdc0b4360 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:27.857 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe43e6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fdc09dec0 con 0x7f6fdc09c870 2024-03-29T11:19:27.857 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe43e6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fdc09da30 con 0x7f6fdc0b4360 2024-03-29T11:19:27.857 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe43e6640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fdc0b39c0 con 0x7f6fdc09ac40 2024-03-29T11:19:27.857 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe43e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:27.857 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe43e6640 10 monclient(hunting): authenticate will time out at 823.143757s 2024-03-29T11:19:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe2983640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f6fdc0b4360 0x7f6fdc14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37736/0 (socket says 172.21.15.82:37736) 2024-03-29T11:19:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe2983640 1 -- 172.21.15.82:0/3028162081 learned_addr learned my addr 172.21.15.82:0/3028162081 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe1180640 1 -- 172.21.15.82:0/3028162081 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700357896 0 0) 0x7f6fdc09da30 con 0x7f6fdc0b4360 2024-03-29T11:19:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe1180640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe1180640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe1180640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe1180640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:19:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe1180640 1 -- 172.21.15.82:0/3028162081 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc4003520 con 0x7f6fdc0b4360 2024-03-29T11:19:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe1180640 1 -- 172.21.15.82:0/3028162081 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185984674 0 0) 0x7f6fdc0b39c0 con 0x7f6fdc09ac40 2024-03-29T11:19:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe1180640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe1180640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe1180640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.858 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe1180640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.872+0000 7f6fe1180640 1 -- 172.21.15.82:0/3028162081 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fdc09da30 con 0x7f6fdc09ac40 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 1 -- 172.21.15.82:0/3028162081 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667750703 0 0) 0x7f6fd0001110 con 0x7f6fdc09c870 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 1 -- 172.21.15.82:0/3028162081 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fdc0b39c0 con 0x7f6fdc09c870 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 1 -- 172.21.15.82:0/3028162081 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1352344281 0 0) 0x7f6fc4003520 con 0x7f6fdc0b4360 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 1 -- 172.21.15.82:0/3028162081 >> v1:172.21.15.67:6790/0 conn(0x7f6fdc09ac40 legacy=0x7f6fdc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 1 -- 172.21.15.82:0/3028162081 >> v1:172.21.15.67:6789/0 conn(0x7f6fdc09c870 legacy=0x7f6fdc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient: _finish_hunting 0 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 1 monclient: found mon.noname-c 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:19:27.859 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 1 -- 172.21.15.82:0/3028162081 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc0b5160 con 0x7f6fdc0b4360 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient: _finish_auth 0 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient: _check_auth_tickets 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 1 -- 172.21.15.82:0/3028162081 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6fcc001a10 con 0x7f6fdc0b4360 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe43e6640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe43e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient: dump: 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.860 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.861 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:27.861 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:27.861 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:27.861 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.861 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe43e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:27.861 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 1 -- 172.21.15.82:0/3028162081 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fcc001bf0 con 0x7f6fdc0b4360 2024-03-29T11:19:27.861 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:27.861 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 1 -- 172.21.15.82:0/3028162081 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6fcc002000 con 0x7f6fdc0b4360 2024-03-29T11:19:27.861 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe43e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:27.861 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe43e6640 4 set_mon_vals no callback set 2024-03-29T11:19:27.861 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.861 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:27.861 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe1180640 10 monclient: dump: 2024-03-29T11:19:27.861 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:27.862 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:27.862 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.862 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.862 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.862 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.862 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:27.862 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:27.862 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:27.862 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.862 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe43e6640 10 monclient: shutdown 2024-03-29T11:19:27.862 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe43e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:27.862 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe43e6640 1 -- 172.21.15.82:0/3028162081 >> v1:172.21.15.82:6789/0 conn(0x7f6fdc0b4360 legacy=0x7f6fdc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.862 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe43e6640 1 -- 172.21.15.82:0/3028162081 shutdown_connections 2024-03-29T11:19:27.863 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe43e6640 1 -- 172.21.15.82:0/3028162081 >> 172.21.15.82:0/3028162081 conn(0x7f6fdc066b00 msgr2=0x7f6fdc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:27.863 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe43e6640 1 -- 172.21.15.82:0/3028162081 shutdown_connections 2024-03-29T11:19:27.863 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.873+0000 7f6fe43e6640 1 -- 172.21.15.82:0/3028162081 wait complete. 2024-03-29T11:19:27.863 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.874+0000 7f6fe43e6640 10 monclient: build_initial_monmap 2024-03-29T11:19:27.863 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.874+0000 7f6fe43e6640 10 monclient: monmap: 2024-03-29T11:19:27.863 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:27.863 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:27.863 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T11:19:27.863 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T11:19:27.863 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:27.863 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.864 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:27.864 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:27.864 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:27.864 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.864 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.874+0000 7f6fe43e6640 1 librados: starting msgr at 2024-03-29T11:19:27.864 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.874+0000 7f6fe43e6640 1 librados: starting objecter 2024-03-29T11:19:27.864 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.874+0000 7f6fe43e6640 1 Processor -- start 2024-03-29T11:19:27.864 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.874+0000 7f6fe43e6640 1 -- start start 2024-03-29T11:19:27.864 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.874+0000 7f6fe43e6640 1 librados: setting wanted keys 2024-03-29T11:19:27.864 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.874+0000 7f6fe43e6640 1 librados: calling monclient init 2024-03-29T11:19:27.864 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.874+0000 7f6fe43e6640 10 monclient: init 2024-03-29T11:19:27.864 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fe43e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:27.864 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fe43e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:27.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fe43e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6fdc09ac40 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:27.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fe43e6640 10 monclient(hunting): picked mon.noname-b con 0x7f6fdc1d8690 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:27.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fe43e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6fdc1455c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:27.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fe43e6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fdc1d7af0 con 0x7f6fdc1455c0 2024-03-29T11:19:27.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fe43e6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fdc145a10 con 0x7f6fdc1d8690 2024-03-29T11:19:27.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fe43e6640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fdc0b5a20 con 0x7f6fdc09ac40 2024-03-29T11:19:27.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fe43e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:27.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fe43e6640 10 monclient(hunting): authenticate will time out at 823.146588s 2024-03-29T11:19:27.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fe1981640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f6fdc1d8690 0x7f6fdc144eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37750/0 (socket says 172.21.15.82:37750) 2024-03-29T11:19:27.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fe1981640 1 -- 172.21.15.82:0/595454323 learned_addr learned my addr 172.21.15.82:0/595454323 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:27.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 client.?.objecter ms_handle_connect 0x7f6fdc1d8690 2024-03-29T11:19:27.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:27.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 client.?.objecter ms_handle_connect 0x7f6fdc09ac40 2024-03-29T11:19:27.865 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:27.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 client.?.objecter ms_handle_connect 0x7f6fdc1455c0 2024-03-29T11:19:27.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:27.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561576777 0 0) 0x7f6fdc145a10 con 0x7f6fdc1d8690 2024-03-29T11:19:27.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:19:27.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc8004eb0 con 0x7f6fdc1d8690 2024-03-29T11:19:27.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226611400 0 0) 0x7f6fdc0b5a20 con 0x7f6fdc09ac40 2024-03-29T11:19:27.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.866 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:19:27.867 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fdc145a10 con 0x7f6fdc09ac40 2024-03-29T11:19:27.867 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415808334 0 0) 0x7f6fdc1d7af0 con 0x7f6fdc1455c0 2024-03-29T11:19:27.867 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.867 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.867 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.867 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:19:27.867 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fdc0b5a20 con 0x7f6fdc1455c0 2024-03-29T11:19:27.867 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3175691550 0 0) 0x7f6fc8004eb0 con 0x7f6fdc1d8690 2024-03-29T11:19:27.867 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fdc1d7af0 con 0x7f6fdc1d8690 2024-03-29T11:19:27.867 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6fd0001860 con 0x7f6fdc1d8690 2024-03-29T11:19:27.867 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.867 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:27.867 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): dump: 2024-03-29T11:19:27.867 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3736834382 0 0) 0x7f6fdc145a10 con 0x7f6fdc09ac40 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fc8004eb0 con 0x7f6fdc09ac40 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6fd4001930 con 0x7f6fdc09ac40 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.868 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:27.869 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.875+0000 7f6fdaffd640 10 monclient(hunting): dump: 2024-03-29T11:19:27.869 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:27.869 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:27.869 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.869 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.869 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.869 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.869 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:27.869 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:27.869 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:27.869 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.869 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 685244022 0 0) 0x7f6fdc0b5a20 con 0x7f6fdc1455c0 2024-03-29T11:19:27.869 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fdc145a10 con 0x7f6fdc1455c0 2024-03-29T11:19:27.870 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6fcc001f00 con 0x7f6fdc1455c0 2024-03-29T11:19:27.870 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.870 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:27.870 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 10 monclient(hunting): dump: 2024-03-29T11:19:27.870 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:27.870 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:27.870 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.870 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.870 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.870 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.870 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:27.870 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:27.870 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:27.870 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1300582842 0 0) 0x7f6fdc1d7af0 con 0x7f6fdc1d8690 2024-03-29T11:19:27.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 >> v1:172.21.15.67:6790/0 conn(0x7f6fdc09ac40 legacy=0x7f6fdc1d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 >> v1:172.21.15.67:6789/0 conn(0x7f6fdc1455c0 legacy=0x7f6fdc1e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:27.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 1 monclient: found mon.b 2024-03-29T11:19:27.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:27.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:27.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc141960 con 0x7f6fdc1d8690 2024-03-29T11:19:27.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:27.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:27.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:27.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fe43e6640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:19:27.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fe43e6640 10 monclient: _renew_subs 2024-03-29T11:19:27.871 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fe43e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:27.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fe43e6640 1 -- 172.21.15.82:0/595454323 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fdc1e3b20 con 0x7f6fdc1d8690 2024-03-29T11:19:27.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fe43e6640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:27.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fe43e6640 10 monclient: _renew_subs 2024-03-29T11:19:27.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fe43e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:27.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fe43e6640 1 -- 172.21.15.82:0/595454323 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fdc1e3e90 con 0x7f6fdc1d8690 2024-03-29T11:19:27.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fe43e6640 1 librados: init done 2024-03-29T11:19:27.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fd0001d50 con 0x7f6fdc1d8690 2024-03-29T11:19:27.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:27.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6fd00037b0 con 0x7f6fdc1d8690 2024-03-29T11:19:27.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.872 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 10 monclient: dump: 2024-03-29T11:19:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:27.873 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fe097f640 4 set_mon_vals no callback set 2024-03-29T11:19:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6fd0003ca0 con 0x7f6fdc1d8690 2024-03-29T11:19:27.873 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 856473697 0 0) 0x7f6fd0006430 con 0x7f6fdc1d8690 2024-03-29T11:19:27.874 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 10 client.4117.objecter ms_dispatch 0x7f6fdc000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:19:27.874 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:19:27.874 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:19:27.874 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.876+0000 7f6fdaffd640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T11:19:27.874 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.877+0000 7f6fe43e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:27.874 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.877+0000 7f6fe43e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:27.874 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.877+0000 7f6fe43e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:27.874 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.877+0000 7f6fe43e6640 1 -- 172.21.15.82:0/595454323 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fdc09b020 con 0x7f6fdc1d8690 2024-03-29T11:19:27.874 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.878+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6fd00057c0 con 0x7f6fdc1d8690 2024-03-29T11:19:27.874 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.878+0000 7f6fdaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:27.874 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.878+0000 7f6fdaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:27.929 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.951+0000 7f6fe43e6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0269e563-f168-4fd0-9bf0-b38c6a7fb937", "id": 1}] 2024-03-29T11:19:27.929 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.951+0000 7f6fe43e6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0269e563-f168-4fd0-9bf0-b38c6a7fb937", "id": 1}] 2024-03-29T11:19:27.929 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.951+0000 7f6fe43e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:27.929 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.951+0000 7f6fe43e6640 1 -- 172.21.15.82:0/595454323 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0269e563-f168-4fd0-9bf0-b38c6a7fb937", "id": 1} v 0) v1 -- 0x7f6fdc143280 con 0x7f6fdc1d8690 2024-03-29T11:19:27.936 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.960+0000 7f6fdaffd640 1 -- 172.21.15.82:0/595454323 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0269e563-f168-4fd0-9bf0-b38c6a7fb937", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 4257210500 0 940483276) 0x7f6fd0005a20 con 0x7f6fdc1d8690 2024-03-29T11:19:27.937 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.960+0000 7f6fdaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0269e563-f168-4fd0-9bf0-b38c6a7fb937", "id": 1}] 2024-03-29T11:19:27.937 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.960+0000 7f6fdaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:27.937 INFO:teuthology.orchestra.run.smithi082.stdout:1 2024-03-29T11:19:27.938 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.960+0000 7f6fe43e6640 10 librados: watch_flush enter 2024-03-29T11:19:27.938 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.961+0000 7f6fe43e6640 10 librados: watch_flush exit 2024-03-29T11:19:27.938 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.961+0000 7f6fe43e6640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T11:19:27.938 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.961+0000 7f6fe43e6640 10 client.4117.objecter successfully canceled tick 2024-03-29T11:19:27.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.961+0000 7f6fe43e6640 10 monclient: shutdown 2024-03-29T11:19:27.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.961+0000 7f6fe43e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:27.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.961+0000 7f6fe43e6640 1 -- 172.21.15.82:0/595454323 >> v1:172.21.15.82:6789/0 conn(0x7f6fdc1d8690 legacy=0x7f6fdc144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.961+0000 7f6fe43e6640 1 -- 172.21.15.82:0/595454323 shutdown_connections 2024-03-29T11:19:27.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.961+0000 7f6fe43e6640 1 -- 172.21.15.82:0/595454323 >> 172.21.15.82:0/595454323 conn(0x7f6fdc066b00 msgr2=0x7f6fdc13a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:27.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.961+0000 7f6fe43e6640 1 -- 172.21.15.82:0/595454323 shutdown_connections 2024-03-29T11:19:27.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.961+0000 7f6fe43e6640 1 -- 172.21.15.82:0/595454323 wait complete. 2024-03-29T11:19:27.939 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:27.961+0000 7f6fe43e6640 1 librados: shutdown 2024-03-29T11:19:27.948 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 701716de-70a2-4029-b6a7-7e8dfe75f51b 2 2024-03-29T11:19:28.048 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.071+0000 7fc73f6a8640 10 monclient: get_monmap_and_config 2024-03-29T11:19:28.048 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.071+0000 7fc73f6a8640 10 monclient: build_initial_monmap 2024-03-29T11:19:28.048 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.071+0000 7fc73f6a8640 10 monclient: monmap: 2024-03-29T11:19:28.048 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:28.048 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.048 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:19:28.072732+0000 2024-03-29T11:19:28.048 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:19:28.072732+0000 2024-03-29T11:19:28.049 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:28.049 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.049 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:28.049 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:28.049 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:28.049 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.058 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.072+0000 7fc73f6a8640 1 Processor -- start 2024-03-29T11:19:28.058 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.072+0000 7fc73f6a8640 1 -- start start 2024-03-29T11:19:28.058 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.072+0000 7fc73f6a8640 10 monclient: init 2024-03-29T11:19:28.058 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.072+0000 7fc73f6a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:28.058 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.072+0000 7fc73f6a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:28.058 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.072+0000 7fc73f6a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc7380b1780 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:28.058 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.072+0000 7fc73f6a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc73813d7e0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:28.058 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc73f6a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc738143be0 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:28.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc73f6a8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc738070b30 con 0x7fc738143be0 2024-03-29T11:19:28.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc73f6a8640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc738070da0 con 0x7fc73813d7e0 2024-03-29T11:19:28.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc73f6a8640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73809adb0 con 0x7fc7380b1780 2024-03-29T11:19:28.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc73f6a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:28.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc73f6a8640 10 monclient(hunting): authenticate will time out at 823.344274s 2024-03-29T11:19:28.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc73cc43640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fc73813d7e0 0x7fc7381434d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37754/0 (socket says 172.21.15.82:37754) 2024-03-29T11:19:28.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc73cc43640 1 -- 172.21.15.82:0/1568013701 learned_addr learned my addr 172.21.15.82:0/1568013701 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:28.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 1 -- 172.21.15.82:0/1568013701 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62367943 0 0) 0x7fc738070da0 con 0x7fc73813d7e0 2024-03-29T11:19:28.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:19:28.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 1 -- 172.21.15.82:0/1568013701 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc724003520 con 0x7fc73813d7e0 2024-03-29T11:19:28.059 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 1 -- 172.21.15.82:0/1568013701 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953754130 0 0) 0x7fc73809adb0 con 0x7fc7380b1780 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 1 -- 172.21.15.82:0/1568013701 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc738070da0 con 0x7fc7380b1780 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 1 -- 172.21.15.82:0/1568013701 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435670686 0 0) 0x7fc738070b30 con 0x7fc738143be0 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 1 -- 172.21.15.82:0/1568013701 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc73809adb0 con 0x7fc738143be0 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 1 -- 172.21.15.82:0/1568013701 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2798149262 0 0) 0x7fc724003520 con 0x7fc73813d7e0 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 1 -- 172.21.15.82:0/1568013701 >> v1:172.21.15.67:6790/0 conn(0x7fc7380b1780 legacy=0x7fc73813d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 1 -- 172.21.15.82:0/1568013701 >> v1:172.21.15.67:6789/0 conn(0x7fc738143be0 legacy=0x7fc73814b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.060 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:28.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 1 monclient: found mon.noname-c 2024-03-29T11:19:28.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:28.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 1 -- 172.21.15.82:0/1568013701 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc738144070 con 0x7fc73813d7e0 2024-03-29T11:19:28.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient: _finish_auth 0 2024-03-29T11:19:28.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:28.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:28.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 1 -- 172.21.15.82:0/1568013701 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc728001690 con 0x7fc73813d7e0 2024-03-29T11:19:28.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:28.061 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient: dump: 2024-03-29T11:19:28.061 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.061 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.061 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.062 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.062 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.062 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.062 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.062 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.062 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.062 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.062 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc73f6a8640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:19:28.062 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc73f6a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:28.062 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 1 -- 172.21.15.82:0/1568013701 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7280017f0 con 0x7fc73813d7e0 2024-03-29T11:19:28.062 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:28.062 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 1 -- 172.21.15.82:0/1568013701 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc728001c00 con 0x7fc73813d7e0 2024-03-29T11:19:28.062 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc73f6a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:28.063 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc73f6a8640 4 set_mon_vals no callback set 2024-03-29T11:19:28.063 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.063 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:28.063 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.073+0000 7fc72ffff640 10 monclient: dump: 2024-03-29T11:19:28.063 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.063 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.063 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.063 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.063 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.063 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.063 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.063 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.063 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.063 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.074+0000 7fc73f6a8640 10 monclient: shutdown 2024-03-29T11:19:28.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.074+0000 7fc73f6a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:28.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.074+0000 7fc73f6a8640 1 -- 172.21.15.82:0/1568013701 >> v1:172.21.15.82:6789/0 conn(0x7fc73813d7e0 legacy=0x7fc7381434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.074+0000 7fc73f6a8640 1 -- 172.21.15.82:0/1568013701 shutdown_connections 2024-03-29T11:19:28.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.074+0000 7fc73f6a8640 1 -- 172.21.15.82:0/1568013701 >> 172.21.15.82:0/1568013701 conn(0x7fc7380b58f0 msgr2=0x7fc7380b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:28.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.074+0000 7fc73f6a8640 1 -- 172.21.15.82:0/1568013701 shutdown_connections 2024-03-29T11:19:28.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.074+0000 7fc73f6a8640 1 -- 172.21.15.82:0/1568013701 wait complete. 2024-03-29T11:19:28.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.074+0000 7fc73f6a8640 10 monclient: build_initial_monmap 2024-03-29T11:19:28.064 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.074+0000 7fc73f6a8640 10 monclient: monmap: 2024-03-29T11:19:28.064 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:28.064 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:28.064 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T11:19:28.065 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T11:19:28.065 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:28.065 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.065 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:28.065 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:28.065 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:28.065 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 1 librados: starting msgr at 2024-03-29T11:19:28.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 1 librados: starting objecter 2024-03-29T11:19:28.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 1 Processor -- start 2024-03-29T11:19:28.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 1 -- start start 2024-03-29T11:19:28.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 1 librados: setting wanted keys 2024-03-29T11:19:28.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 1 librados: calling monclient init 2024-03-29T11:19:28.065 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 10 monclient: init 2024-03-29T11:19:28.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:28.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:28.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc7380b1780 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:28.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc73813d7e0 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:28.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc738143be0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:28.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc738139530 con 0x7fc73813d7e0 2024-03-29T11:19:28.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7381428c0 con 0x7fc738143be0 2024-03-29T11:19:28.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc738142a50 con 0x7fc7380b1780 2024-03-29T11:19:28.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:28.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73f6a8640 10 monclient(hunting): authenticate will time out at 823.347082s 2024-03-29T11:19:28.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73dc45640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fc738143be0 0x7fc7381da3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37764/0 (socket says 172.21.15.82:37764) 2024-03-29T11:19:28.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc73dc45640 1 -- 172.21.15.82:0/417209423 learned_addr learned my addr 172.21.15.82:0/417209423 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:28.066 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.075+0000 7fc72dffb640 10 client.?.objecter ms_handle_connect 0x7fc738143be0 2024-03-29T11:19:28.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 client.?.objecter ms_handle_connect 0x7fc7380b1780 2024-03-29T11:19:28.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 client.?.objecter ms_handle_connect 0x7fc73813d7e0 2024-03-29T11:19:28.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699934283 0 0) 0x7fc7381428c0 con 0x7fc738143be0 2024-03-29T11:19:28.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:19:28.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc71c004eb0 con 0x7fc738143be0 2024-03-29T11:19:28.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984304868 0 0) 0x7fc738142a50 con 0x7fc7380b1780 2024-03-29T11:19:28.067 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.068 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.068 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.068 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:19:28.068 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7381428c0 con 0x7fc7380b1780 2024-03-29T11:19:28.068 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463948658 0 0) 0x7fc738139530 con 0x7fc73813d7e0 2024-03-29T11:19:28.068 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.068 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.068 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.068 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:19:28.068 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc738142a50 con 0x7fc73813d7e0 2024-03-29T11:19:28.068 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 39993448 0 0) 0x7fc71c004eb0 con 0x7fc738143be0 2024-03-29T11:19:28.068 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc738139530 con 0x7fc738143be0 2024-03-29T11:19:28.068 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc730001be0 con 0x7fc738143be0 2024-03-29T11:19:28.068 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): dump: 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1994905454 0 0) 0x7fc7381428c0 con 0x7fc7380b1780 2024-03-29T11:19:28.069 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc71c004eb0 con 0x7fc7380b1780 2024-03-29T11:19:28.070 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc7340018f0 con 0x7fc7380b1780 2024-03-29T11:19:28.070 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.070 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:28.070 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): dump: 2024-03-29T11:19:28.070 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.070 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.070 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.070 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.070 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.070 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.070 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.070 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.070 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.070 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2838628250 0 0) 0x7fc738142a50 con 0x7fc73813d7e0 2024-03-29T11:19:28.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7381428c0 con 0x7fc73813d7e0 2024-03-29T11:19:28.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc728001b00 con 0x7fc73813d7e0 2024-03-29T11:19:28.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:28.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient(hunting): dump: 2024-03-29T11:19:28.071 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.071 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.071 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.071 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.071 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.071 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.071 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.072 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.072 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.072 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.072 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1919107951 0 0) 0x7fc738139530 con 0x7fc738143be0 2024-03-29T11:19:28.072 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 >> v1:172.21.15.67:6790/0 conn(0x7fc7380b1780 legacy=0x7fc738138cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.072 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 >> v1:172.21.15.67:6789/0 conn(0x7fc73813d7e0 legacy=0x7fc738135bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.072 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:28.072 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 monclient: found mon.b 2024-03-29T11:19:28.072 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:28.072 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.072 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc738139e40 con 0x7fc738143be0 2024-03-29T11:19:28.072 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:28.072 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:28.072 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc73f6a8640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc73f6a8640 10 monclient: _renew_subs 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc73f6a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc73f6a8640 1 -- 172.21.15.82:0/417209423 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73813a090 con 0x7fc738143be0 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc730003990 con 0x7fc738143be0 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc730003d60 con 0x7fc738143be0 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 monclient: dump: 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.073 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.074 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.074 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.074 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.074 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.074 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.074 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72f7fe640 4 set_mon_vals no callback set 2024-03-29T11:19:28.074 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc73f6a8640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:28.074 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc73f6a8640 10 monclient: _renew_subs 2024-03-29T11:19:28.074 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc7300042d0 con 0x7fc738143be0 2024-03-29T11:19:28.074 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc73f6a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.074 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc73f6a8640 1 -- 172.21.15.82:0/417209423 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7381364c0 con 0x7fc738143be0 2024-03-29T11:19:28.074 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc73f6a8640 1 librados: init done 2024-03-29T11:19:28.074 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2117927011 0 0) 0x7fc730004580 con 0x7fc738143be0 2024-03-29T11:19:28.075 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 10 client.4123.objecter ms_dispatch 0x7fc738000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:19:28.075 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:19:28.075 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.076+0000 7fc72dffb640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:19:28.075 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.077+0000 7fc72dffb640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T11:19:28.075 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.077+0000 7fc73f6a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.075 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.077+0000 7fc73f6a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.075 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.077+0000 7fc73f6a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.075 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.077+0000 7fc73f6a8640 1 -- 172.21.15.82:0/417209423 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc738067c00 con 0x7fc738143be0 2024-03-29T11:19:28.075 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.079+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc730004740 con 0x7fc738143be0 2024-03-29T11:19:28.075 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.079+0000 7fc72dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.075 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.079+0000 7fc72dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:28.091 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:28.115+0000 7f0c9364b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:28.092 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:28.115+0000 7fcb6aa38040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:28.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.151+0000 7fc73f6a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "701716de-70a2-4029-b6a7-7e8dfe75f51b", "id": 2}] 2024-03-29T11:19:28.131 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.151+0000 7fc73f6a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "701716de-70a2-4029-b6a7-7e8dfe75f51b", "id": 2}] 2024-03-29T11:19:28.131 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.151+0000 7fc73f6a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.131 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.151+0000 7fc73f6a8640 1 -- 172.21.15.82:0/417209423 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd new", "uuid": "701716de-70a2-4029-b6a7-7e8dfe75f51b", "id": 2} v 0) v1 -- 0x7fc73804f730 con 0x7fc738143be0 2024-03-29T11:19:28.133 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.157+0000 7fc72dffb640 1 -- 172.21.15.82:0/417209423 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "701716de-70a2-4029-b6a7-7e8dfe75f51b", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1260760971 0 216599125) 0x7fc730005b40 con 0x7fc738143be0 2024-03-29T11:19:28.134 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.157+0000 7fc72dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "701716de-70a2-4029-b6a7-7e8dfe75f51b", "id": 2}] 2024-03-29T11:19:28.134 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.157+0000 7fc72dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:28.134 INFO:teuthology.orchestra.run.smithi082.stdout:2 2024-03-29T11:19:28.135 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.158+0000 7fc73f6a8640 10 librados: watch_flush enter 2024-03-29T11:19:28.136 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.158+0000 7fc73f6a8640 10 librados: watch_flush exit 2024-03-29T11:19:28.136 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.158+0000 7fc73f6a8640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T11:19:28.136 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.158+0000 7fc73f6a8640 10 client.4123.objecter successfully canceled tick 2024-03-29T11:19:28.136 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.158+0000 7fc73f6a8640 10 monclient: shutdown 2024-03-29T11:19:28.136 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.158+0000 7fc73f6a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:28.136 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.158+0000 7fc73f6a8640 1 -- 172.21.15.82:0/417209423 >> v1:172.21.15.82:6789/0 conn(0x7fc738143be0 legacy=0x7fc7381da3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.136 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.158+0000 7fc73f6a8640 1 -- 172.21.15.82:0/417209423 shutdown_connections 2024-03-29T11:19:28.136 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.158+0000 7fc73f6a8640 1 -- 172.21.15.82:0/417209423 >> 172.21.15.82:0/417209423 conn(0x7fc7380b58f0 msgr2=0x7fc7380b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:28.136 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.158+0000 7fc73f6a8640 1 -- 172.21.15.82:0/417209423 shutdown_connections 2024-03-29T11:19:28.136 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.158+0000 7fc73f6a8640 1 -- 172.21.15.82:0/417209423 wait complete. 2024-03-29T11:19:28.136 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.158+0000 7fc73f6a8640 1 librados: shutdown 2024-03-29T11:19:28.145 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new c6b7be97-5c3e-4789-8d61-d0f4ae124b04 3 2024-03-29T11:19:28.148 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:28.172+0000 7f0c9364b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:28.149 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:28.172+0000 7fcb6aa38040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:28.195 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:28.219+0000 7f0c9364b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:28.197 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:28.220+0000 7fcb6aa38040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:28.244 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.268+0000 7fd3632c1640 10 monclient: get_monmap_and_config 2024-03-29T11:19:28.245 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.268+0000 7fd3632c1640 10 monclient: build_initial_monmap 2024-03-29T11:19:28.245 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.268+0000 7fd3632c1640 10 monclient: monmap: 2024-03-29T11:19:28.245 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:28.245 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.245 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:19:28.269198+0000 2024-03-29T11:19:28.245 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:19:28.269198+0000 2024-03-29T11:19:28.245 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:28.245 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.245 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:28.245 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:28.245 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:28.245 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.253 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.268+0000 7fd3632c1640 1 Processor -- start 2024-03-29T11:19:28.253 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.268+0000 7fd3632c1640 1 -- start start 2024-03-29T11:19:28.253 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.268+0000 7fd3632c1640 10 monclient: init 2024-03-29T11:19:28.253 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd3632c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:28.253 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd3632c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:28.253 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd3632c1640 10 monclient(hunting): picked mon.noname-b con 0x7fd35c09ac40 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:28.253 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd3632c1640 10 monclient(hunting): picked mon.noname-c con 0x7fd35c09c870 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:28.253 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd3632c1640 10 monclient(hunting): picked mon.noname-a con 0x7fd35c0b4360 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:28.253 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd3632c1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c09dec0 con 0x7fd35c0b4360 2024-03-29T11:19:28.253 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd3632c1640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c09da30 con 0x7fd35c09c870 2024-03-29T11:19:28.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd3632c1640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c0b39c0 con 0x7fd35c09ac40 2024-03-29T11:19:28.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd3632c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:28.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd3632c1640 10 monclient(hunting): authenticate will time out at 823.540802s 2024-03-29T11:19:28.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd36085c640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fd35c09c870 0x7fd35c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37772/0 (socket says 172.21.15.82:37772) 2024-03-29T11:19:28.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd36085c640 1 -- 172.21.15.82:0/993924014 learned_addr learned my addr 172.21.15.82:0/993924014 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:28.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd353fff640 1 -- 172.21.15.82:0/993924014 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129282509 0 0) 0x7fd35c09da30 con 0x7fd35c09c870 2024-03-29T11:19:28.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd353fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd353fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd353fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd353fff640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:19:28.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.269+0000 7fd353fff640 1 -- 172.21.15.82:0/993924014 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd348003520 con 0x7fd35c09c870 2024-03-29T11:19:28.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 1 -- 172.21.15.82:0/993924014 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115215055 0 0) 0x7fd354001490 con 0x7fd35c0b4360 2024-03-29T11:19:28.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:19:28.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 1 -- 172.21.15.82:0/993924014 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd35c09da30 con 0x7fd35c0b4360 2024-03-29T11:19:28.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 1 -- 172.21.15.82:0/993924014 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264037523 0 0) 0x7fd35c0b39c0 con 0x7fd35c09ac40 2024-03-29T11:19:28.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:19:28.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 1 -- 172.21.15.82:0/993924014 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd354001490 con 0x7fd35c09ac40 2024-03-29T11:19:28.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 1 -- 172.21.15.82:0/993924014 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 885411374 0 0) 0x7fd348003520 con 0x7fd35c09c870 2024-03-29T11:19:28.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 1 -- 172.21.15.82:0/993924014 >> v1:172.21.15.67:6790/0 conn(0x7fd35c09ac40 legacy=0x7fd35c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 1 -- 172.21.15.82:0/993924014 >> v1:172.21.15.67:6789/0 conn(0x7fd35c0b4360 legacy=0x7fd35c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:28.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 1 monclient: found mon.noname-c 2024-03-29T11:19:28.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:28.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 1 -- 172.21.15.82:0/993924014 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c0b5160 con 0x7fd35c09c870 2024-03-29T11:19:28.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:28.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:28.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:28.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 1 -- 172.21.15.82:0/993924014 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd358001690 con 0x7fd35c09c870 2024-03-29T11:19:28.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd3632c1640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:19:28.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd3632c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:28.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.257 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:28.257 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient: dump: 2024-03-29T11:19:28.257 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.257 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.257 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.257 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.257 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.257 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.257 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.257 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.257 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.257 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.257 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd3632c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:28.257 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 1 -- 172.21.15.82:0/993924014 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd358001870 con 0x7fd35c09c870 2024-03-29T11:19:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 1 -- 172.21.15.82:0/993924014 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd358001c80 con 0x7fd35c09c870 2024-03-29T11:19:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd353fff640 10 monclient: dump: 2024-03-29T11:19:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.258 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.259 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.259 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd3632c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:28.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd3632c1640 4 set_mon_vals no callback set 2024-03-29T11:19:28.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd3632c1640 10 monclient: shutdown 2024-03-29T11:19:28.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd3632c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:28.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd3632c1640 1 -- 172.21.15.82:0/993924014 >> v1:172.21.15.82:6789/0 conn(0x7fd35c09c870 legacy=0x7fd35c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd3632c1640 1 -- 172.21.15.82:0/993924014 shutdown_connections 2024-03-29T11:19:28.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd3632c1640 1 -- 172.21.15.82:0/993924014 >> 172.21.15.82:0/993924014 conn(0x7fd35c066b00 msgr2=0x7fd35c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:28.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd3632c1640 1 -- 172.21.15.82:0/993924014 shutdown_connections 2024-03-29T11:19:28.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.270+0000 7fd3632c1640 1 -- 172.21.15.82:0/993924014 wait complete. 2024-03-29T11:19:28.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.271+0000 7fd3632c1640 10 monclient: build_initial_monmap 2024-03-29T11:19:28.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.271+0000 7fd3632c1640 10 monclient: monmap: 2024-03-29T11:19:28.259 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:28.260 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:28.260 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T11:19:28.260 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T11:19:28.260 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:28.260 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.260 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:28.260 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:28.260 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:28.260 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.260 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.271+0000 7fd3632c1640 1 librados: starting msgr at 2024-03-29T11:19:28.260 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.271+0000 7fd3632c1640 1 librados: starting objecter 2024-03-29T11:19:28.260 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.271+0000 7fd3632c1640 1 Processor -- start 2024-03-29T11:19:28.260 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.271+0000 7fd3632c1640 1 -- start start 2024-03-29T11:19:28.261 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.271+0000 7fd3632c1640 1 librados: setting wanted keys 2024-03-29T11:19:28.261 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.271+0000 7fd3632c1640 1 librados: calling monclient init 2024-03-29T11:19:28.261 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.271+0000 7fd3632c1640 10 monclient: init 2024-03-29T11:19:28.261 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd3632c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:28.261 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd3632c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:28.261 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd3632c1640 10 monclient(hunting): picked mon.noname-a con 0x7fd35c09ac40 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:28.261 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd3632c1640 10 monclient(hunting): picked mon.noname-c con 0x7fd35c0b6b00 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:28.261 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd3632c1640 10 monclient(hunting): picked mon.noname-b con 0x7fd35c1406b0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:28.261 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd3632c1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c1d7e20 con 0x7fd35c09ac40 2024-03-29T11:19:28.261 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd3632c1640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c140b00 con 0x7fd35c1406b0 2024-03-29T11:19:28.261 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd3632c1640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c0ad080 con 0x7fd35c0b6b00 2024-03-29T11:19:28.261 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd3632c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:28.261 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd3632c1640 10 monclient(hunting): authenticate will time out at 823.543731s 2024-03-29T11:19:28.261 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd36185e640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fd35c1406b0 0x7fd35c1e1e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37780/0 (socket says 172.21.15.82:37780) 2024-03-29T11:19:28.262 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd36185e640 1 -- 172.21.15.82:0/152170111 learned_addr learned my addr 172.21.15.82:0/152170111 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:28.262 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 client.?.objecter ms_handle_connect 0x7fd35c1406b0 2024-03-29T11:19:28.262 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.262 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 client.?.objecter ms_handle_connect 0x7fd35c09ac40 2024-03-29T11:19:28.262 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.262 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 client.?.objecter ms_handle_connect 0x7fd35c0b6b00 2024-03-29T11:19:28.262 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.262 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642959336 0 0) 0x7fd35c140b00 con 0x7fd35c1406b0 2024-03-29T11:19:28.262 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.262 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.262 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.262 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:19:28.262 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd340004eb0 con 0x7fd35c1406b0 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296511102 0 0) 0x7fd35c1d7e20 con 0x7fd35c09ac40 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd35c140b00 con 0x7fd35c09ac40 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285747082 0 0) 0x7fd35c0ad080 con 0x7fd35c0b6b00 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.272+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd35c1d7e20 con 0x7fd35c0b6b00 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 361480523 0 0) 0x7fd340004eb0 con 0x7fd35c1406b0 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd35c0ad080 con 0x7fd35c1406b0 2024-03-29T11:19:28.263 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd354001c80 con 0x7fd35c1406b0 2024-03-29T11:19:28.264 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.264 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:28.264 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:28.264 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.264 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.264 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.264 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.264 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.264 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.264 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.264 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.264 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.264 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.264 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:28.265 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1482074733 0 0) 0x7fd35c140b00 con 0x7fd35c09ac40 2024-03-29T11:19:28.265 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd340004eb0 con 0x7fd35c09ac40 2024-03-29T11:19:28.265 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd34c0018b0 con 0x7fd35c09ac40 2024-03-29T11:19:28.265 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.265 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:28.265 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:28.265 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.265 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.265 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.265 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.265 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.265 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.265 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.266 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.266 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.266 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.266 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 918920953 0 0) 0x7fd35c1d7e20 con 0x7fd35c0b6b00 2024-03-29T11:19:28.266 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd35c140b00 con 0x7fd35c0b6b00 2024-03-29T11:19:28.266 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd358001b80 con 0x7fd35c0b6b00 2024-03-29T11:19:28.266 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.266 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:28.266 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:28.266 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.266 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.266 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.266 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.267 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.267 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.267 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.267 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.267 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.267 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.267 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2683226500 0 0) 0x7fd35c0ad080 con 0x7fd35c1406b0 2024-03-29T11:19:28.267 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 >> v1:172.21.15.67:6790/0 conn(0x7fd35c0b6b00 legacy=0x7fd35c13ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.267 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 >> v1:172.21.15.67:6789/0 conn(0x7fd35c09ac40 legacy=0x7fd35c1d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.267 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:28.267 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 monclient: found mon.b 2024-03-29T11:19:28.267 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:28.267 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.267 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c1d7ab0 con 0x7fd35c1406b0 2024-03-29T11:19:28.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:28.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:28.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:28.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd3632c1640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:19:28.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd3632c1640 10 monclient: _renew_subs 2024-03-29T11:19:28.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd3632c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd3632c1640 1 -- 172.21.15.82:0/152170111 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c1e3570 con 0x7fd35c1406b0 2024-03-29T11:19:28.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd354003970 con 0x7fd35c1406b0 2024-03-29T11:19:28.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:28.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd354003d80 con 0x7fd35c1406b0 2024-03-29T11:19:28.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:28.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 monclient: dump: 2024-03-29T11:19:28.268 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd3537fe640 4 set_mon_vals no callback set 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd3632c1640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd3540042f0 con 0x7fd35c1406b0 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd3632c1640 10 monclient: _renew_subs 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd3632c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.269 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd3632c1640 1 -- 172.21.15.82:0/152170111 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c1e38e0 con 0x7fd35c1406b0 2024-03-29T11:19:28.270 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd3632c1640 1 librados: init done 2024-03-29T11:19:28.270 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3479072792 0 0) 0x7fd3540047d0 con 0x7fd35c1406b0 2024-03-29T11:19:28.270 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 10 client.4129.objecter ms_dispatch 0x7fd35c000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:19:28.270 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:19:28.270 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:19:28.270 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.273+0000 7fd351ffb640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T11:19:28.270 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.274+0000 7fd3632c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.270 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.274+0000 7fd3632c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.270 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.274+0000 7fd3632c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.270 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.274+0000 7fd3632c1640 1 -- 172.21.15.82:0/152170111 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35c09b020 con 0x7fd35c1406b0 2024-03-29T11:19:28.270 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.276+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd354005b40 con 0x7fd35c1406b0 2024-03-29T11:19:28.271 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.276+0000 7fd351ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.271 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.276+0000 7fd351ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:28.271 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:28.277+0000 7fcb6aa38040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:28.271 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:28.275+0000 7f0c9364b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:28.296 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:28.320+0000 7f0c9364b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:28.298 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:28.322+0000 7fcb6aa38040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:28.326 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.349+0000 7fd3632c1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c6b7be97-5c3e-4789-8d61-d0f4ae124b04", "id": 3}] 2024-03-29T11:19:28.326 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.349+0000 7fd3632c1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c6b7be97-5c3e-4789-8d61-d0f4ae124b04", "id": 3}] 2024-03-29T11:19:28.326 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.349+0000 7fd3632c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.326 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.349+0000 7fd3632c1640 1 -- 172.21.15.82:0/152170111 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c6b7be97-5c3e-4789-8d61-d0f4ae124b04", "id": 3} v 0) v1 -- 0x7fd35c04f730 con 0x7fd35c1406b0 2024-03-29T11:19:28.331 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.354+0000 7fd351ffb640 1 -- 172.21.15.82:0/152170111 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c6b7be97-5c3e-4789-8d61-d0f4ae124b04", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 617361697 0 525046306) 0x7fd354005d60 con 0x7fd35c1406b0 2024-03-29T11:19:28.331 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.354+0000 7fd351ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c6b7be97-5c3e-4789-8d61-d0f4ae124b04", "id": 3}] 2024-03-29T11:19:28.331 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.354+0000 7fd351ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:28.331 INFO:teuthology.orchestra.run.smithi082.stdout:3 2024-03-29T11:19:28.332 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.355+0000 7fd3632c1640 10 librados: watch_flush enter 2024-03-29T11:19:28.333 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.355+0000 7fd3632c1640 10 librados: watch_flush exit 2024-03-29T11:19:28.333 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.355+0000 7fd3632c1640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T11:19:28.333 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.355+0000 7fd3632c1640 10 client.4129.objecter successfully canceled tick 2024-03-29T11:19:28.333 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.355+0000 7fd3632c1640 10 monclient: shutdown 2024-03-29T11:19:28.333 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.355+0000 7fd3632c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:28.333 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.355+0000 7fd3632c1640 1 -- 172.21.15.82:0/152170111 >> v1:172.21.15.82:6789/0 conn(0x7fd35c1406b0 legacy=0x7fd35c1e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.333 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.355+0000 7fd3632c1640 1 -- 172.21.15.82:0/152170111 shutdown_connections 2024-03-29T11:19:28.333 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.355+0000 7fd3632c1640 1 -- 172.21.15.82:0/152170111 >> 172.21.15.82:0/152170111 conn(0x7fd35c066b00 msgr2=0x7fd35c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:28.333 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.355+0000 7fd3632c1640 1 -- 172.21.15.82:0/152170111 shutdown_connections 2024-03-29T11:19:28.333 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.355+0000 7fd3632c1640 1 -- 172.21.15.82:0/152170111 wait complete. 2024-03-29T11:19:28.333 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.355+0000 7fd3632c1640 1 librados: shutdown 2024-03-29T11:19:28.342 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new f0dcbb47-e60b-4ee5-aa81-29812af8118a 4 2024-03-29T11:19:28.417 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:28.440+0000 7f0c9364b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:28.420 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:28.443+0000 7fcb6aa38040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:28.482 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.505+0000 7f30a7093640 10 monclient: get_monmap_and_config 2024-03-29T11:19:28.483 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.505+0000 7f30a7093640 10 monclient: build_initial_monmap 2024-03-29T11:19:28.483 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.506+0000 7f30a7093640 10 monclient: monmap: 2024-03-29T11:19:28.483 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:28.483 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.483 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:19:28.507069+0000 2024-03-29T11:19:28.483 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:19:28.507069+0000 2024-03-29T11:19:28.483 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:28.483 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.483 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:28.483 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:28.483 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:28.483 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.506+0000 7f30a7093640 1 Processor -- start 2024-03-29T11:19:28.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.506+0000 7f30a7093640 1 -- start start 2024-03-29T11:19:28.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.506+0000 7f30a7093640 10 monclient: init 2024-03-29T11:19:28.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f30a7093640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:28.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f30a7093640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:28.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f30a7093640 10 monclient(hunting): picked mon.noname-a con 0x7f30a009ac40 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:28.493 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f30a7093640 10 monclient(hunting): picked mon.noname-c con 0x7f30a009c870 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:28.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f30a7093640 10 monclient(hunting): picked mon.noname-b con 0x7f30a00b4360 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:28.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f30a7093640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a009dec0 con 0x7f30a009ac40 2024-03-29T11:19:28.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f30a7093640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a009da30 con 0x7f30a009c870 2024-03-29T11:19:28.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f30a7093640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a00b39c0 con 0x7f30a00b4360 2024-03-29T11:19:28.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f30a7093640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:28.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f30a7093640 10 monclient(hunting): authenticate will time out at 823.778642s 2024-03-29T11:19:28.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309ffff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f30a009c870 0x7f30a013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37790/0 (socket says 172.21.15.82:37790) 2024-03-29T11:19:28.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309ffff640 1 -- 172.21.15.82:0/978460895 learned_addr learned my addr 172.21.15.82:0/978460895 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:28.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 1 -- 172.21.15.82:0/978460895 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103052956 0 0) 0x7f30a009da30 con 0x7f30a009c870 2024-03-29T11:19:28.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.494 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.495 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:19:28.495 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 1 -- 172.21.15.82:0/978460895 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3090003520 con 0x7f30a009c870 2024-03-29T11:19:28.495 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 1 -- 172.21.15.82:0/978460895 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915935897 0 0) 0x7f3094001110 con 0x7f30a009ac40 2024-03-29T11:19:28.495 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.495 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.495 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.495 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:19:28.495 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 1 -- 172.21.15.82:0/978460895 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a009da30 con 0x7f30a009ac40 2024-03-29T11:19:28.495 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 1 -- 172.21.15.82:0/978460895 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447760450 0 0) 0x7f30a00b39c0 con 0x7f30a00b4360 2024-03-29T11:19:28.495 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.495 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.495 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.495 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:19:28.496 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.507+0000 7f309f7fe640 1 -- 172.21.15.82:0/978460895 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3094001110 con 0x7f30a00b4360 2024-03-29T11:19:28.496 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 1 -- 172.21.15.82:0/978460895 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2217936018 0 0) 0x7f3090003520 con 0x7f30a009c870 2024-03-29T11:19:28.496 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 1 -- 172.21.15.82:0/978460895 >> v1:172.21.15.67:6790/0 conn(0x7f30a00b4360 legacy=0x7f30a014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.496 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 1 -- 172.21.15.82:0/978460895 >> v1:172.21.15.67:6789/0 conn(0x7f30a009ac40 legacy=0x7f30a00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.496 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:28.496 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:19:28.496 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:28.496 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.496 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 1 -- 172.21.15.82:0/978460895 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a00b5160 con 0x7f30a009c870 2024-03-29T11:19:28.497 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:28.497 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:28.497 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:28.497 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 1 -- 172.21.15.82:0/978460895 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f308c001690 con 0x7f30a009c870 2024-03-29T11:19:28.497 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f30a7093640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:19:28.497 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f30a7093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:28.497 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.497 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:28.497 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 10 monclient: dump: 2024-03-29T11:19:28.497 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.497 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.497 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.498 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.498 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.498 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.498 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.498 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.498 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.498 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.498 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f30a7093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:28.498 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 1 -- 172.21.15.82:0/978460895 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f308c001870 con 0x7f30a009c870 2024-03-29T11:19:28.498 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:28.498 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 1 -- 172.21.15.82:0/978460895 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f308c001cc0 con 0x7f30a009c870 2024-03-29T11:19:28.498 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f30a7093640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:28.499 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f30a7093640 4 set_mon_vals no callback set 2024-03-29T11:19:28.499 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.499 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:28.499 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f309f7fe640 10 monclient: dump: 2024-03-29T11:19:28.499 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.499 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.499 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.499 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.499 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.499 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.499 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.499 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.499 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.499 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f30a7093640 10 monclient: shutdown 2024-03-29T11:19:28.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f30a7093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:28.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f30a7093640 1 -- 172.21.15.82:0/978460895 >> v1:172.21.15.82:6789/0 conn(0x7f30a009c870 legacy=0x7f30a013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f30a7093640 1 -- 172.21.15.82:0/978460895 shutdown_connections 2024-03-29T11:19:28.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f30a7093640 1 -- 172.21.15.82:0/978460895 >> 172.21.15.82:0/978460895 conn(0x7f30a0066b00 msgr2=0x7f30a0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:28.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f30a7093640 1 -- 172.21.15.82:0/978460895 shutdown_connections 2024-03-29T11:19:28.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.508+0000 7f30a7093640 1 -- 172.21.15.82:0/978460895 wait complete. 2024-03-29T11:19:28.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.509+0000 7f30a7093640 10 monclient: build_initial_monmap 2024-03-29T11:19:28.500 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.509+0000 7f30a7093640 10 monclient: monmap: 2024-03-29T11:19:28.500 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:28.500 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:28.500 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T11:19:28.500 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T11:19:28.501 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:28.501 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.501 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:28.501 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:28.501 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:28.501 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.501 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.509+0000 7f30a7093640 1 librados: starting msgr at 2024-03-29T11:19:28.501 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.509+0000 7f30a7093640 1 librados: starting objecter 2024-03-29T11:19:28.501 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.509+0000 7f30a7093640 1 Processor -- start 2024-03-29T11:19:28.501 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.509+0000 7f30a7093640 1 -- start start 2024-03-29T11:19:28.501 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.509+0000 7f30a7093640 1 librados: setting wanted keys 2024-03-29T11:19:28.501 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.509+0000 7f30a7093640 1 librados: calling monclient init 2024-03-29T11:19:28.501 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.509+0000 7f30a7093640 10 monclient: init 2024-03-29T11:19:28.501 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f30a7093640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:28.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f30a7093640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:28.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f30a7093640 10 monclient(hunting): picked mon.noname-c con 0x7f30a009ac40 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:28.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f30a7093640 10 monclient(hunting): picked mon.noname-b con 0x7f30a00acd00 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:28.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f30a7093640 10 monclient(hunting): picked mon.noname-a con 0x7f30a0142d30 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:28.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f30a7093640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a013bc10 con 0x7f30a0142d30 2024-03-29T11:19:28.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f30a7093640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a013bed0 con 0x7f30a00acd00 2024-03-29T11:19:28.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f30a7093640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a00ae950 con 0x7f30a009ac40 2024-03-29T11:19:28.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f30a7093640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:28.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f30a7093640 10 monclient(hunting): authenticate will time out at 823.781798s 2024-03-29T11:19:28.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309ffff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f30a00acd00 0x7f30a0141350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37794/0 (socket says 172.21.15.82:37794) 2024-03-29T11:19:28.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309ffff640 1 -- 172.21.15.82:0/1047189794 learned_addr learned my addr 172.21.15.82:0/1047189794 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:28.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 10 client.?.objecter ms_handle_connect 0x7f30a00acd00 2024-03-29T11:19:28.502 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 10 client.?.objecter ms_handle_connect 0x7f30a009ac40 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 10 client.?.objecter ms_handle_connect 0x7f30a0142d30 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785292887 0 0) 0x7f30a013bed0 con 0x7f30a00acd00 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3084004eb0 con 0x7f30a00acd00 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412833421 0 0) 0x7f30a013bc10 con 0x7f30a0142d30 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:19:28.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a013bed0 con 0x7f30a0142d30 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.510+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129270021 0 0) 0x7f30a00ae950 con 0x7f30a009ac40 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a013bc10 con 0x7f30a009ac40 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 985834159 0 0) 0x7f3084004eb0 con 0x7f30a00acd00 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30a00ae950 con 0x7f30a00acd00 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f308c001bc0 con 0x7f30a00acd00 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.504 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2268511914 0 0) 0x7f30a013bed0 con 0x7f30a0142d30 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3084004eb0 con 0x7f30a0142d30 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f3098001a70 con 0x7f30a0142d30 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.505 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.506 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.506 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.506 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.506 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.506 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.506 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.506 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.506 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1847843801 0 0) 0x7f30a013bc10 con 0x7f30a009ac40 2024-03-29T11:19:28.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30a013bed0 con 0x7f30a009ac40 2024-03-29T11:19:28.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3060613567 0 0) 0x7f30a00ae950 con 0x7f30a00acd00 2024-03-29T11:19:28.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 >> v1:172.21.15.67:6790/0 conn(0x7f30a009ac40 legacy=0x7f30a00af0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 >> v1:172.21.15.67:6789/0 conn(0x7f30a0142d30 legacy=0x7f30a01dab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:28.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 monclient: found mon.b 2024-03-29T11:19:28.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:28.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a00aec30 con 0x7f30a00acd00 2024-03-29T11:19:28.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:28.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:28.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:28.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f30a7093640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:19:28.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f30a7093640 10 monclient: _renew_subs 2024-03-29T11:19:28.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f30a7093640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f30a7093640 1 -- 172.21.15.82:0/1047189794 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a00af7d0 con 0x7f30a00acd00 2024-03-29T11:19:28.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f30a7093640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:28.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f30a7093640 10 monclient: _renew_subs 2024-03-29T11:19:28.507 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f30a7093640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f30a7093640 1 -- 172.21.15.82:0/1047189794 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a01db470 con 0x7f30a00acd00 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f30a7093640 1 librados: init done 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f308c004bf0 con 0x7f30a00acd00 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f308c004f00 con 0x7f30a00acd00 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309effd640 4 set_mon_vals no callback set 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 monclient: dump: 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.508 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.509 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.509 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.509 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f308c0053f0 con 0x7f30a00acd00 2024-03-29T11:19:28.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 782599718 0 0) 0x7f308c0069f0 con 0x7f30a00acd00 2024-03-29T11:19:28.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 10 client.4135.objecter ms_dispatch 0x7f30a0000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:19:28.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:19:28.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:19:28.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.511+0000 7f309d7fa640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T11:19:28.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.512+0000 7f30a7093640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.512+0000 7f30a7093640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.512+0000 7f30a7093640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.512+0000 7f30a7093640 1 -- 172.21.15.82:0/1047189794 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3080005080 con 0x7f30a00acd00 2024-03-29T11:19:28.509 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.515+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f308c0064e0 con 0x7f30a00acd00 2024-03-29T11:19:28.510 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.515+0000 7f309d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.510 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.515+0000 7f309d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:28.510 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:28.519+0000 7fcb6aa38040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:28.510 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:28.514+0000 7f0c9364b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:28.542 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:28.562+0000 7f0c9364b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:28.544 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:28.567+0000 7fcb6aa38040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:28.565 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.588+0000 7f30a7093640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f0dcbb47-e60b-4ee5-aa81-29812af8118a", "id": 4}] 2024-03-29T11:19:28.565 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.588+0000 7f30a7093640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f0dcbb47-e60b-4ee5-aa81-29812af8118a", "id": 4}] 2024-03-29T11:19:28.565 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.588+0000 7f30a7093640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.565 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.588+0000 7f30a7093640 1 -- 172.21.15.82:0/1047189794 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f0dcbb47-e60b-4ee5-aa81-29812af8118a", "id": 4} v 0) v1 -- 0x7f30800058b0 con 0x7f30a00acd00 2024-03-29T11:19:28.570 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.594+0000 7f309d7fa640 1 -- 172.21.15.82:0/1047189794 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f0dcbb47-e60b-4ee5-aa81-29812af8118a", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3818668020 0 1697012583) 0x7f308c006830 con 0x7f30a00acd00 2024-03-29T11:19:28.571 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.594+0000 7f309d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f0dcbb47-e60b-4ee5-aa81-29812af8118a", "id": 4}] 2024-03-29T11:19:28.571 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.594+0000 7f309d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:28.571 INFO:teuthology.orchestra.run.smithi082.stdout:4 2024-03-29T11:19:28.572 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.595+0000 7f30a7093640 10 librados: watch_flush enter 2024-03-29T11:19:28.572 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.595+0000 7f30a7093640 10 librados: watch_flush exit 2024-03-29T11:19:28.572 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.595+0000 7f30a7093640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T11:19:28.572 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.595+0000 7f30a7093640 10 client.4135.objecter successfully canceled tick 2024-03-29T11:19:28.573 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.595+0000 7f30a7093640 10 monclient: shutdown 2024-03-29T11:19:28.573 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.595+0000 7f30a7093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:28.573 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.595+0000 7f30a7093640 1 -- 172.21.15.82:0/1047189794 >> v1:172.21.15.82:6789/0 conn(0x7f30a00acd00 legacy=0x7f30a0141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.573 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.595+0000 7f30a7093640 1 -- 172.21.15.82:0/1047189794 shutdown_connections 2024-03-29T11:19:28.573 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.595+0000 7f30a7093640 1 -- 172.21.15.82:0/1047189794 >> 172.21.15.82:0/1047189794 conn(0x7f30a0066b00 msgr2=0x7f30a013cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:28.573 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.595+0000 7f30a7093640 1 -- 172.21.15.82:0/1047189794 shutdown_connections 2024-03-29T11:19:28.573 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.595+0000 7f30a7093640 1 -- 172.21.15.82:0/1047189794 wait complete. 2024-03-29T11:19:28.573 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.595+0000 7f30a7093640 1 librados: shutdown 2024-03-29T11:19:28.582 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new e557d6a6-20ec-4600-a31d-f553df8334b8 5 2024-03-29T11:19:28.594 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:28.618+0000 7f0c9364b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:28.600 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:28.623+0000 7fcb6aa38040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:28.643 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:28.667+0000 7f0c9364b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:28.650 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:28.673+0000 7fcb6aa38040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:28.698 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.721+0000 7f141d923640 10 monclient: get_monmap_and_config 2024-03-29T11:19:28.698 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.721+0000 7f141d923640 10 monclient: build_initial_monmap 2024-03-29T11:19:28.698 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.721+0000 7f141d923640 10 monclient: monmap: 2024-03-29T11:19:28.698 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:28.698 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.699 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:19:28.722905+0000 2024-03-29T11:19:28.699 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:19:28.722905+0000 2024-03-29T11:19:28.699 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:28.699 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.699 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:28.699 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:28.699 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:28.699 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.722+0000 7f141d923640 1 Processor -- start 2024-03-29T11:19:28.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.722+0000 7f141d923640 1 -- start start 2024-03-29T11:19:28.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.722+0000 7f141d923640 10 monclient: init 2024-03-29T11:19:28.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.722+0000 7f141d923640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:28.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.722+0000 7f141d923640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:28.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f141d923640 10 monclient(hunting): picked mon.noname-c con 0x7f14180b8240 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:28.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f141d923640 10 monclient(hunting): picked mon.noname-a con 0x7f14180b1f60 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:28.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f141d923640 10 monclient(hunting): picked mon.noname-b con 0x7f141809c8c0 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:28.708 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f141d923640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f141809bd80 con 0x7f14180b1f60 2024-03-29T11:19:28.709 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f141d923640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f141809b960 con 0x7f14180b8240 2024-03-29T11:19:28.709 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f141d923640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14180b2ec0 con 0x7f141809c8c0 2024-03-29T11:19:28.709 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f141d923640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:28.709 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f141d923640 10 monclient(hunting): authenticate will time out at 823.994313s 2024-03-29T11:19:28.709 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1416ffd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f14180b8240 0x7f141813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37802/0 (socket says 172.21.15.82:37802) 2024-03-29T11:19:28.709 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1416ffd640 1 -- 172.21.15.82:0/2260381066 learned_addr learned my addr 172.21.15.82:0/2260381066 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:28.709 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 1 -- 172.21.15.82:0/2260381066 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039860190 0 0) 0x7f141809b960 con 0x7f14180b8240 2024-03-29T11:19:28.709 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.709 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.709 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.709 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:19:28.709 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 1 -- 172.21.15.82:0/2260381066 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1400003520 con 0x7f14180b8240 2024-03-29T11:19:28.709 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 1 -- 172.21.15.82:0/2260381066 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709427920 0 0) 0x7f141809bd80 con 0x7f14180b1f60 2024-03-29T11:19:28.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:19:28.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 1 -- 172.21.15.82:0/2260381066 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f141809b960 con 0x7f14180b1f60 2024-03-29T11:19:28.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 1 -- 172.21.15.82:0/2260381066 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787088492 0 0) 0x7f14180b2ec0 con 0x7f141809c8c0 2024-03-29T11:19:28.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:19:28.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 1 -- 172.21.15.82:0/2260381066 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f141809bd80 con 0x7f141809c8c0 2024-03-29T11:19:28.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 1 -- 172.21.15.82:0/2260381066 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2029434259 0 0) 0x7f1400003520 con 0x7f14180b8240 2024-03-29T11:19:28.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 1 -- 172.21.15.82:0/2260381066 >> v1:172.21.15.67:6790/0 conn(0x7f141809c8c0 legacy=0x7f141814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 1 -- 172.21.15.82:0/2260381066 >> v1:172.21.15.67:6789/0 conn(0x7f14180b1f60 legacy=0x7f141809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:28.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:28.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:28.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 1 -- 172.21.15.82:0/2260381066 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141813fb60 con 0x7f14180b8240 2024-03-29T11:19:28.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:28.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:28.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:28.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 1 -- 172.21.15.82:0/2260381066 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f1408001690 con 0x7f14180b8240 2024-03-29T11:19:28.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f141d923640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:19:28.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f141d923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:28.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:28.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient: dump: 2024-03-29T11:19:28.712 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.712 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.712 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.712 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.712 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.712 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.712 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.712 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.712 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.712 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f141d923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:28.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 1 -- 172.21.15.82:0/2260381066 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1408001870 con 0x7f14180b8240 2024-03-29T11:19:28.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:28.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 1 -- 172.21.15.82:0/2260381066 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f1408001c80 con 0x7f14180b8240 2024-03-29T11:19:28.713 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f141d923640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:28.713 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f141d923640 4 set_mon_vals no callback set 2024-03-29T11:19:28.713 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.713 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:28.713 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.723+0000 7f1415ffb640 10 monclient: dump: 2024-03-29T11:19:28.713 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.713 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.713 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.713 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.713 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.713 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.713 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.713 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.713 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.714 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.714 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.724+0000 7f141d923640 10 monclient: shutdown 2024-03-29T11:19:28.714 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.724+0000 7f141d923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:28.714 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.724+0000 7f141d923640 1 -- 172.21.15.82:0/2260381066 >> v1:172.21.15.82:6789/0 conn(0x7f14180b8240 legacy=0x7f141813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.714 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.724+0000 7f141d923640 1 -- 172.21.15.82:0/2260381066 shutdown_connections 2024-03-29T11:19:28.714 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.724+0000 7f141d923640 1 -- 172.21.15.82:0/2260381066 >> 172.21.15.82:0/2260381066 conn(0x7f1418066b00 msgr2=0x7f1418066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:28.714 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.724+0000 7f141d923640 1 -- 172.21.15.82:0/2260381066 shutdown_connections 2024-03-29T11:19:28.714 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.724+0000 7f141d923640 1 -- 172.21.15.82:0/2260381066 wait complete. 2024-03-29T11:19:28.714 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.724+0000 7f141d923640 10 monclient: build_initial_monmap 2024-03-29T11:19:28.714 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.724+0000 7f141d923640 10 monclient: monmap: 2024-03-29T11:19:28.714 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:28.714 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:28.714 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.725+0000 7f141d923640 1 librados: starting msgr at 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.725+0000 7f141d923640 1 librados: starting objecter 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.725+0000 7f141d923640 1 Processor -- start 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.725+0000 7f141d923640 1 -- start start 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.725+0000 7f141d923640 1 librados: setting wanted keys 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.725+0000 7f141d923640 1 librados: calling monclient init 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.725+0000 7f141d923640 10 monclient: init 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:28.715 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 10 monclient(hunting): picked mon.noname-a con 0x7f141809c8c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 10 monclient(hunting): picked mon.noname-c con 0x7f14180b8240 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 10 monclient(hunting): picked mon.noname-b con 0x7f14181d3e50 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14181461a0 con 0x7f141809c8c0 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14180aff90 con 0x7f14181d3e50 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14180b0200 con 0x7f14180b8240 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 10 monclient(hunting): authenticate will time out at 823.997369s 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f14177fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f14181d3e50 0x7f14181d4490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37814/0 (socket says 172.21.15.82:37814) 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f14177fe640 1 -- 172.21.15.82:0/2415598568 learned_addr learned my addr 172.21.15.82:0/2415598568 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 client.?.objecter ms_handle_connect 0x7f14181d3e50 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 client.?.objecter ms_handle_connect 0x7f141809c8c0 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.716 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 client.?.objecter ms_handle_connect 0x7f14180b8240 2024-03-29T11:19:28.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113476863 0 0) 0x7f14180aff90 con 0x7f14181d3e50 2024-03-29T11:19:28.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:19:28.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1404004eb0 con 0x7f14181d3e50 2024-03-29T11:19:28.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528306986 0 0) 0x7f14181461a0 con 0x7f141809c8c0 2024-03-29T11:19:28.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:19:28.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14180aff90 con 0x7f141809c8c0 2024-03-29T11:19:28.717 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098739005 0 0) 0x7f14180b0200 con 0x7f14180b8240 2024-03-29T11:19:28.718 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.718 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.718 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.718 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:19:28.718 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14181461a0 con 0x7f14180b8240 2024-03-29T11:19:28.718 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 448712929 0 0) 0x7f1404004eb0 con 0x7f14181d3e50 2024-03-29T11:19:28.718 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14180b0200 con 0x7f14181d3e50 2024-03-29T11:19:28.718 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f1410001c70 con 0x7f14181d3e50 2024-03-29T11:19:28.718 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.718 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:28.718 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): dump: 2024-03-29T11:19:28.718 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.718 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.718 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.719 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.719 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.719 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.719 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.719 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.719 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.719 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.719 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:28.719 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3351160887 0 0) 0x7f14180aff90 con 0x7f141809c8c0 2024-03-29T11:19:28.719 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1404004eb0 con 0x7f141809c8c0 2024-03-29T11:19:28.719 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f1408001b80 con 0x7f141809c8c0 2024-03-29T11:19:28.719 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.719 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:28.719 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): dump: 2024-03-29T11:19:28.720 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.720 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.720 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.720 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.720 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.720 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.720 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.720 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.720 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.720 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2350273322 0 0) 0x7f14181461a0 con 0x7f14180b8240 2024-03-29T11:19:28.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14180aff90 con 0x7f14180b8240 2024-03-29T11:19:28.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f140c001750 con 0x7f14180b8240 2024-03-29T11:19:28.721 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.721 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:28.721 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient(hunting): dump: 2024-03-29T11:19:28.721 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.721 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.721 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.721 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.721 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.721 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.721 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.721 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.721 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.721 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.721 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1890402382 0 0) 0x7f14180b0200 con 0x7f14181d3e50 2024-03-29T11:19:28.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 >> v1:172.21.15.67:6790/0 conn(0x7f14180b8240 legacy=0x7f14180ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 >> v1:172.21.15.67:6789/0 conn(0x7f141809c8c0 legacy=0x7f14181d79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:28.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 monclient: found mon.b 2024-03-29T11:19:28.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:28.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14181d7480 con 0x7f14181d3e50 2024-03-29T11:19:28.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient: _finish_auth 0 2024-03-29T11:19:28.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:28.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f13fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:28.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:19:28.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 10 monclient: _renew_subs 2024-03-29T11:19:28.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 1 -- 172.21.15.82:0/2415598568 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14181d8110 con 0x7f14181d3e50 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.726+0000 7f141d923640 10 monclient: _renew_subs 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f141d923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f141d923640 1 -- 172.21.15.82:0/2415598568 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14180aee00 con 0x7f14181d3e50 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f141d923640 1 librados: init done 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1410003990 con 0x7f14181d3e50 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f13fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f1410003da0 con 0x7f14181d3e50 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f13fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f13fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f14157fa640 4 set_mon_vals no callback set 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f13fffff640 10 monclient: dump: 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.723 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1410004350 con 0x7f14181d3e50 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1685082768 0 0) 0x7f1410006b20 con 0x7f14181d3e50 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f13fffff640 10 client.4141.objecter ms_dispatch 0x7f1418000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f13fffff640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f13fffff640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f13fffff640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f141d923640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.724 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f141d923640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.725 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f141d923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.725 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.727+0000 7f141d923640 1 -- 172.21.15.82:0/2415598568 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f8005080 con 0x7f14181d3e50 2024-03-29T11:19:28.725 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.729+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f14100065a0 con 0x7f14181d3e50 2024-03-29T11:19:28.725 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.729+0000 7f13fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.725 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.729+0000 7f13fffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:28.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.800+0000 7f141d923640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e557d6a6-20ec-4600-a31d-f553df8334b8", "id": 5}] 2024-03-29T11:19:28.777 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.800+0000 7f141d923640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e557d6a6-20ec-4600-a31d-f553df8334b8", "id": 5}] 2024-03-29T11:19:28.777 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.800+0000 7f141d923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.777 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.800+0000 7f141d923640 1 -- 172.21.15.82:0/2415598568 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e557d6a6-20ec-4600-a31d-f553df8334b8", "id": 5} v 0) v1 -- 0x7f13f80058b0 con 0x7f14181d3e50 2024-03-29T11:19:28.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.807+0000 7f13fffff640 1 -- 172.21.15.82:0/2415598568 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e557d6a6-20ec-4600-a31d-f553df8334b8", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1141536243 0 1988412176) 0x7f14100067c0 con 0x7f14181d3e50 2024-03-29T11:19:28.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.807+0000 7f13fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e557d6a6-20ec-4600-a31d-f553df8334b8", "id": 5}] 2024-03-29T11:19:28.784 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.807+0000 7f13fffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:28.784 INFO:teuthology.orchestra.run.smithi082.stdout:5 2024-03-29T11:19:28.785 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.808+0000 7f141d923640 10 librados: watch_flush enter 2024-03-29T11:19:28.786 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.808+0000 7f141d923640 10 librados: watch_flush exit 2024-03-29T11:19:28.786 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.808+0000 7f141d923640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:19:28.786 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.808+0000 7f141d923640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:19:28.786 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.808+0000 7f141d923640 10 monclient: shutdown 2024-03-29T11:19:28.786 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.808+0000 7f141d923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:28.786 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.808+0000 7f141d923640 1 -- 172.21.15.82:0/2415598568 >> v1:172.21.15.82:6789/0 conn(0x7f14181d3e50 legacy=0x7f14181d4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.786 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.808+0000 7f141d923640 1 -- 172.21.15.82:0/2415598568 shutdown_connections 2024-03-29T11:19:28.786 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.808+0000 7f141d923640 1 -- 172.21.15.82:0/2415598568 >> 172.21.15.82:0/2415598568 conn(0x7f1418066b00 msgr2=0x7f141813dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:28.786 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.808+0000 7f141d923640 1 -- 172.21.15.82:0/2415598568 shutdown_connections 2024-03-29T11:19:28.786 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.808+0000 7f141d923640 1 -- 172.21.15.82:0/2415598568 wait complete. 2024-03-29T11:19:28.786 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.808+0000 7f141d923640 1 librados: shutdown 2024-03-29T11:19:28.795 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 9c9039c9-2b83-4f2c-8820-b5f2dd9fad50 6 2024-03-29T11:19:28.895 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.918+0000 7f5bbd70e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:28.895 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.918+0000 7f5bbd70e640 10 monclient: build_initial_monmap 2024-03-29T11:19:28.895 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.918+0000 7f5bbd70e640 10 monclient: monmap: 2024-03-29T11:19:28.895 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:28.895 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.896 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:19:28.919631+0000 2024-03-29T11:19:28.896 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:19:28.919631+0000 2024-03-29T11:19:28.896 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:28.896 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.896 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:28.896 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:28.896 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:28.896 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.906 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bbd70e640 1 Processor -- start 2024-03-29T11:19:28.906 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bbd70e640 1 -- start start 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bbd70e640 10 monclient: init 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bbd70e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bbd70e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bbd70e640 10 monclient(hunting): picked mon.noname-a con 0x7f5bb80b8240 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bbd70e640 10 monclient(hunting): picked mon.noname-b con 0x7f5bb80b1f60 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bbd70e640 10 monclient(hunting): picked mon.noname-c con 0x7f5bb809c8c0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bbd70e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb809bd80 con 0x7f5bb80b8240 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bbd70e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb809b960 con 0x7f5bb809c8c0 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bbd70e640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb80b2ec0 con 0x7f5bb80b1f60 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bbd70e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bbd70e640 10 monclient(hunting): authenticate will time out at 824.191082s 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bb77fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f5bb809c8c0 0x7f5bb814d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37828/0 (socket says 172.21.15.82:37828) 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.919+0000 7f5bb77fe640 1 -- 172.21.15.82:0/4033067694 learned_addr learned my addr 172.21.15.82:0/4033067694 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 1 -- 172.21.15.82:0/4033067694 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937720663 0 0) 0x7f5bb809b960 con 0x7f5bb809c8c0 2024-03-29T11:19:28.907 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 1 -- 172.21.15.82:0/4033067694 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba0003520 con 0x7f5bb809c8c0 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 1 -- 172.21.15.82:0/4033067694 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201356687 0 0) 0x7f5bb80b2ec0 con 0x7f5bb80b1f60 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 1 -- 172.21.15.82:0/4033067694 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb809b960 con 0x7f5bb80b1f60 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 1 -- 172.21.15.82:0/4033067694 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499717587 0 0) 0x7f5bb809bd80 con 0x7f5bb80b8240 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:19:28.908 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 1 -- 172.21.15.82:0/4033067694 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb80b2ec0 con 0x7f5bb80b8240 2024-03-29T11:19:28.909 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 1 -- 172.21.15.82:0/4033067694 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1594999350 0 0) 0x7f5ba0003520 con 0x7f5bb809c8c0 2024-03-29T11:19:28.909 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 1 -- 172.21.15.82:0/4033067694 >> v1:172.21.15.67:6790/0 conn(0x7f5bb80b1f60 legacy=0x7f5bb809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.909 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 1 -- 172.21.15.82:0/4033067694 >> v1:172.21.15.67:6789/0 conn(0x7f5bb80b8240 legacy=0x7f5bb813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.909 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:28.909 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:28.909 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:28.909 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.909 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 1 -- 172.21.15.82:0/4033067694 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb813fb60 con 0x7f5bb809c8c0 2024-03-29T11:19:28.909 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:28.909 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:28.909 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:28.909 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 1 -- 172.21.15.82:0/4033067694 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5bb0001a10 con 0x7f5bb809c8c0 2024-03-29T11:19:28.909 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bbd70e640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:19:28.909 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bbd70e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:28.910 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.910 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:28.910 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient: dump: 2024-03-29T11:19:28.910 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.910 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.910 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.910 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.910 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.910 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.910 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.910 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.910 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.910 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.910 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bbd70e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 1 -- 172.21.15.82:0/4033067694 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bb0001bf0 con 0x7f5bb809c8c0 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bbd70e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 1 -- 172.21.15.82:0/4033067694 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5bb0002000 con 0x7f5bb809c8c0 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bbd70e640 4 set_mon_vals no callback set 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.920+0000 7f5bb5ffb640 10 monclient: dump: 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.911 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.921+0000 7f5bbd70e640 10 monclient: shutdown 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.921+0000 7f5bbd70e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.921+0000 7f5bbd70e640 1 -- 172.21.15.82:0/4033067694 >> v1:172.21.15.82:6789/0 conn(0x7f5bb809c8c0 legacy=0x7f5bb814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.921+0000 7f5bbd70e640 1 -- 172.21.15.82:0/4033067694 shutdown_connections 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.921+0000 7f5bbd70e640 1 -- 172.21.15.82:0/4033067694 >> 172.21.15.82:0/4033067694 conn(0x7f5bb8066b00 msgr2=0x7f5bb8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.921+0000 7f5bbd70e640 1 -- 172.21.15.82:0/4033067694 shutdown_connections 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.921+0000 7f5bbd70e640 1 -- 172.21.15.82:0/4033067694 wait complete. 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.921+0000 7f5bbd70e640 10 monclient: build_initial_monmap 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.921+0000 7f5bbd70e640 10 monclient: monmap: 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:28.912 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.922+0000 7f5bbd70e640 1 librados: starting msgr at 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.922+0000 7f5bbd70e640 1 librados: starting objecter 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.922+0000 7f5bbd70e640 1 Processor -- start 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.922+0000 7f5bbd70e640 1 -- start start 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.922+0000 7f5bbd70e640 1 librados: setting wanted keys 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.922+0000 7f5bbd70e640 1 librados: calling monclient init 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.922+0000 7f5bbd70e640 10 monclient: init 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.922+0000 7f5bbd70e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:28.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.922+0000 7f5bbd70e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.922+0000 7f5bbd70e640 10 monclient(hunting): picked mon.noname-c con 0x7f5bb809c8c0 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 10 monclient(hunting): picked mon.noname-a con 0x7f5bb80b8240 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 10 monclient(hunting): picked mon.noname-b con 0x7f5bb8145490 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb80ae320 con 0x7f5bb80b8240 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb80af530 con 0x7f5bb8145490 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb80af1f0 con 0x7f5bb809c8c0 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 10 monclient(hunting): authenticate will time out at 824.194293s 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bb77fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f5bb8145490 0x7f5bb81da830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37840/0 (socket says 172.21.15.82:37840) 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bb77fe640 1 -- 172.21.15.82:0/2345983541 learned_addr learned my addr 172.21.15.82:0/2345983541 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 client.?.objecter ms_handle_connect 0x7f5bb8145490 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 client.?.objecter ms_handle_connect 0x7f5bb80b8240 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 client.?.objecter ms_handle_connect 0x7f5bb809c8c0 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410700526 0 0) 0x7f5bb80af530 con 0x7f5bb8145490 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba4004eb0 con 0x7f5bb8145490 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189663002 0 0) 0x7f5bb80ae320 con 0x7f5bb80b8240 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb80af530 con 0x7f5bb80b8240 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122988795 0 0) 0x7f5bb80af1f0 con 0x7f5bb809c8c0 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:28.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:28.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:28.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:19:28.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb80ae320 con 0x7f5bb809c8c0 2024-03-29T11:19:28.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 120656645 0 0) 0x7f5ba4004eb0 con 0x7f5bb8145490 2024-03-29T11:19:28.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bb80af1f0 con 0x7f5bb8145490 2024-03-29T11:19:28.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5bb0001f00 con 0x7f5bb8145490 2024-03-29T11:19:28.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:28.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:28.916 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.916 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.916 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.916 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.916 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2077660958 0 0) 0x7f5bb80af530 con 0x7f5bb80b8240 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ba4004eb0 con 0x7f5bb80b8240 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5bac001970 con 0x7f5bb80b8240 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.918 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.918 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.918 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.918 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.918 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.918 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3789996330 0 0) 0x7f5bb80ae320 con 0x7f5bb809c8c0 2024-03-29T11:19:28.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bb80af530 con 0x7f5bb809c8c0 2024-03-29T11:19:28.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5ba80016c0 con 0x7f5bb809c8c0 2024-03-29T11:19:28.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:28.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:28.918 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.918 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4251693328 0 0) 0x7f5bb80af1f0 con 0x7f5bb8145490 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 >> v1:172.21.15.67:6790/0 conn(0x7f5bb809c8c0 legacy=0x7f5bb80aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 >> v1:172.21.15.67:6789/0 conn(0x7f5bb80b8240 legacy=0x7f5bb8134df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 monclient: found mon.b 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb8135500 con 0x7f5bb8145490 2024-03-29T11:19:28.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:28.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:28.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:28.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:19:28.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 10 monclient: _renew_subs 2024-03-29T11:19:28.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 1 -- 172.21.15.82:0/2345983541 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb81357b0 con 0x7f5bb8145490 2024-03-29T11:19:28.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:28.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 10 monclient: _renew_subs 2024-03-29T11:19:28.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 1 -- 172.21.15.82:0/2345983541 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb81db120 con 0x7f5bb8145490 2024-03-29T11:19:28.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5bbd70e640 1 librados: init done 2024-03-29T11:19:28.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.923+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bb0004f70 con 0x7f5bb8145490 2024-03-29T11:19:28.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5b9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:28.921 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5bb0002100 con 0x7f5bb8145490 2024-03-29T11:19:28.921 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5b9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:28.921 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5b9f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:28.921 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5b9f7fe640 10 monclient: dump: 2024-03-29T11:19:28.921 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:28.921 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:28.921 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.921 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:28.921 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:28.921 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:28.921 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:28.921 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:28.921 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:28.921 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:28.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5bb57fa640 4 set_mon_vals no callback set 2024-03-29T11:19:28.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5bb0005610 con 0x7f5bb8145490 2024-03-29T11:19:28.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3675049243 0 0) 0x7f5bb0006e30 con 0x7f5bb8145490 2024-03-29T11:19:28.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5b9f7fe640 10 client.4147.objecter ms_dispatch 0x7f5bb8000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:19:28.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5b9f7fe640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:19:28.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5b9f7fe640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:19:28.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5b9f7fe640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T11:19:28.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5bbd70e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5bbd70e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5bbd70e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.924+0000 7f5bbd70e640 1 -- 172.21.15.82:0/2345983541 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b98005080 con 0x7f5bb8145490 2024-03-29T11:19:28.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.926+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5bb0006840 con 0x7f5bb8145490 2024-03-29T11:19:28.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.926+0000 7f5b9f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:28.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:28.926+0000 7f5b9f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:28.977 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.000+0000 7f5bbd70e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9c9039c9-2b83-4f2c-8820-b5f2dd9fad50", "id": 6}] 2024-03-29T11:19:28.977 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.000+0000 7f5bbd70e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9c9039c9-2b83-4f2c-8820-b5f2dd9fad50", "id": 6}] 2024-03-29T11:19:28.977 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.000+0000 7f5bbd70e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:28.977 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.000+0000 7f5bbd70e640 1 -- 172.21.15.82:0/2345983541 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9c9039c9-2b83-4f2c-8820-b5f2dd9fad50", "id": 6} v 0) v1 -- 0x7f5b980058b0 con 0x7f5bb8145490 2024-03-29T11:19:28.980 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.003+0000 7f5b9f7fe640 1 -- 172.21.15.82:0/2345983541 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9c9039c9-2b83-4f2c-8820-b5f2dd9fad50", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2232588667 0 1113811849) 0x7f5bb0006bb0 con 0x7f5bb8145490 2024-03-29T11:19:28.980 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.003+0000 7f5b9f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9c9039c9-2b83-4f2c-8820-b5f2dd9fad50", "id": 6}] 2024-03-29T11:19:28.980 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.003+0000 7f5b9f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:28.980 INFO:teuthology.orchestra.run.smithi082.stdout:6 2024-03-29T11:19:28.981 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.004+0000 7f5bbd70e640 10 librados: watch_flush enter 2024-03-29T11:19:28.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.004+0000 7f5bbd70e640 10 librados: watch_flush exit 2024-03-29T11:19:28.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.004+0000 7f5bbd70e640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T11:19:28.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.004+0000 7f5bbd70e640 10 client.4147.objecter successfully canceled tick 2024-03-29T11:19:28.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.004+0000 7f5bbd70e640 10 monclient: shutdown 2024-03-29T11:19:28.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.004+0000 7f5bbd70e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:28.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.004+0000 7f5bbd70e640 1 -- 172.21.15.82:0/2345983541 >> v1:172.21.15.82:6789/0 conn(0x7f5bb8145490 legacy=0x7f5bb81da830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:28.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.004+0000 7f5bbd70e640 1 -- 172.21.15.82:0/2345983541 shutdown_connections 2024-03-29T11:19:28.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.004+0000 7f5bbd70e640 1 -- 172.21.15.82:0/2345983541 >> 172.21.15.82:0/2345983541 conn(0x7f5bb8066b00 msgr2=0x7f5bb806bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:28.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.004+0000 7f5bbd70e640 1 -- 172.21.15.82:0/2345983541 shutdown_connections 2024-03-29T11:19:28.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.004+0000 7f5bbd70e640 1 -- 172.21.15.82:0/2345983541 wait complete. 2024-03-29T11:19:28.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.004+0000 7f5bbd70e640 1 librados: shutdown 2024-03-29T11:19:28.991 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new e857aa17-61f7-45e4-824c-d5075c3ebf38 7 2024-03-29T11:19:29.045 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:29.069+0000 7f0c9364b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:29.048 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:29.071+0000 7fcb6aa38040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:29.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.114+0000 7fa0298f1640 10 monclient: get_monmap_and_config 2024-03-29T11:19:29.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.114+0000 7fa0298f1640 10 monclient: build_initial_monmap 2024-03-29T11:19:29.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.114+0000 7fa0298f1640 10 monclient: monmap: 2024-03-29T11:19:29.092 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:29.092 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:29.092 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:19:29.115890+0000 2024-03-29T11:19:29.092 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:19:29.115890+0000 2024-03-29T11:19:29.092 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:29.092 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:29.092 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:29.092 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:29.092 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:29.092 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:29.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.115+0000 7fa0298f1640 1 Processor -- start 2024-03-29T11:19:29.093 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.115+0000 7fa0298f1640 1 -- start start 2024-03-29T11:19:29.093 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.115+0000 7fa0298f1640 10 monclient: init 2024-03-29T11:19:29.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.115+0000 7fa0298f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:29.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.115+0000 7fa0298f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:29.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.115+0000 7fa0298f1640 10 monclient(hunting): picked mon.noname-b con 0x7fa0240b8240 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:29.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.115+0000 7fa0298f1640 10 monclient(hunting): picked mon.noname-c con 0x7fa0240b1f60 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:29.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa0298f1640 10 monclient(hunting): picked mon.noname-a con 0x7fa02409c8c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:29.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa0298f1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02409bd80 con 0x7fa02409c8c0 2024-03-29T11:19:29.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa0298f1640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02409b960 con 0x7fa0240b1f60 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa0298f1640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0240b2ec0 con 0x7fa0240b8240 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa0298f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa0298f1640 10 monclient(hunting): authenticate will time out at 824.387281s 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa0227fc640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fa0240b1f60 0x7fa02409c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37850/0 (socket says 172.21.15.82:37850) 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa0227fc640 1 -- 172.21.15.82:0/3881721646 learned_addr learned my addr 172.21.15.82:0/3881721646 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 1 -- 172.21.15.82:0/3881721646 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374811648 0 0) 0x7fa02409b960 con 0x7fa0240b1f60 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 1 -- 172.21.15.82:0/3881721646 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa010003520 con 0x7fa0240b1f60 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 1 -- 172.21.15.82:0/3881721646 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927687171 0 0) 0x7fa02409bd80 con 0x7fa02409c8c0 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 1 -- 172.21.15.82:0/3881721646 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa02409b960 con 0x7fa02409c8c0 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 1 -- 172.21.15.82:0/3881721646 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735806974 0 0) 0x7fa0240b2ec0 con 0x7fa0240b8240 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 1 -- 172.21.15.82:0/3881721646 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa02409bd80 con 0x7fa0240b8240 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 1 -- 172.21.15.82:0/3881721646 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2378577282 0 0) 0x7fa010003520 con 0x7fa0240b1f60 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 1 -- 172.21.15.82:0/3881721646 >> v1:172.21.15.67:6790/0 conn(0x7fa0240b8240 legacy=0x7fa02413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 1 -- 172.21.15.82:0/3881721646 >> v1:172.21.15.67:6789/0 conn(0x7fa02409c8c0 legacy=0x7fa02414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:19:29.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 1 -- 172.21.15.82:0/3881721646 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02413fb60 con 0x7fa0240b1f60 2024-03-29T11:19:29.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:29.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:29.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:29.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 1 -- 172.21.15.82:0/3881721646 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa00c001690 con 0x7fa0240b1f60 2024-03-29T11:19:29.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa0298f1640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:19:29.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa0298f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:29.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:29.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient: dump: 2024-03-29T11:19:29.102 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:29.102 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:29.102 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.102 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.102 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa0298f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 1 -- 172.21.15.82:0/3881721646 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa00c001870 con 0x7fa0240b1f60 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 1 -- 172.21.15.82:0/3881721646 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa00c001c80 con 0x7fa0240b1f60 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa0298f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa0298f1640 4 set_mon_vals no callback set 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.116+0000 7fa021ffb640 10 monclient: dump: 2024-03-29T11:19:29.103 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.117+0000 7fa0298f1640 10 monclient: shutdown 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.117+0000 7fa0298f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.117+0000 7fa0298f1640 1 -- 172.21.15.82:0/3881721646 >> v1:172.21.15.82:6789/0 conn(0x7fa0240b1f60 legacy=0x7fa02409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.117+0000 7fa0298f1640 1 -- 172.21.15.82:0/3881721646 shutdown_connections 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.117+0000 7fa0298f1640 1 -- 172.21.15.82:0/3881721646 >> 172.21.15.82:0/3881721646 conn(0x7fa024066b00 msgr2=0x7fa024066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:29.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.117+0000 7fa0298f1640 1 -- 172.21.15.82:0/3881721646 shutdown_connections 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.117+0000 7fa0298f1640 1 -- 172.21.15.82:0/3881721646 wait complete. 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.117+0000 7fa0298f1640 10 monclient: build_initial_monmap 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.117+0000 7fa0298f1640 10 monclient: monmap: 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 1 librados: starting msgr at 2024-03-29T11:19:29.105 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 1 librados: starting objecter 2024-03-29T11:19:29.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 1 Processor -- start 2024-03-29T11:19:29.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 1 -- start start 2024-03-29T11:19:29.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 1 librados: setting wanted keys 2024-03-29T11:19:29.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 1 librados: calling monclient init 2024-03-29T11:19:29.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 10 monclient: init 2024-03-29T11:19:29.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:29.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:29.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 10 monclient(hunting): picked mon.noname-a con 0x7fa02409c8c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:29.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 10 monclient(hunting): picked mon.noname-b con 0x7fa0240b8240 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:29.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 10 monclient(hunting): picked mon.noname-c con 0x7fa0240ad9f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:29.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa024133550 con 0x7fa02409c8c0 2024-03-29T11:19:29.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0240ad5d0 con 0x7fa0240b8240 2024-03-29T11:19:29.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa024133c20 con 0x7fa0240ad9f0 2024-03-29T11:19:29.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.118+0000 7fa0298f1640 10 monclient(hunting): authenticate will time out at 824.390224s 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa0227fc640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fa0240b8240 0x7fa024131e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:37852/0 (socket says 172.21.15.82:37852) 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa0227fc640 1 -- 172.21.15.82:0/309530948 learned_addr learned my addr 172.21.15.82:0/309530948 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 client.?.objecter ms_handle_connect 0x7fa0240b8240 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 client.?.objecter ms_handle_connect 0x7fa02409c8c0 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 client.?.objecter ms_handle_connect 0x7fa0240ad9f0 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216331798 0 0) 0x7fa0240ad5d0 con 0x7fa0240b8240 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:19:29.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa004004eb0 con 0x7fa0240b8240 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925241070 0 0) 0x7fa024133550 con 0x7fa02409c8c0 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): my global_id is 4161 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0240ad5d0 con 0x7fa02409c8c0 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281934302 0 0) 0x7fa024133c20 con 0x7fa0240ad9f0 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa024133550 con 0x7fa0240ad9f0 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3553057982 0 0) 0x7fa004004eb0 con 0x7fa0240b8240 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa024133c20 con 0x7fa0240b8240 2024-03-29T11:19:29.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa00c001b80 con 0x7fa0240b8240 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): dump: 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:29.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1103276325 0 0) 0x7fa0240ad5d0 con 0x7fa02409c8c0 2024-03-29T11:19:29.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa004004eb0 con 0x7fa02409c8c0 2024-03-29T11:19:29.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa0140017d0 con 0x7fa02409c8c0 2024-03-29T11:19:29.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:29.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): dump: 2024-03-29T11:19:29.110 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:29.110 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:29.110 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.110 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.110 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.110 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:29.110 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:29.110 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:29.110 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:29.111 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:29.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 741068281 0 0) 0x7fa024133550 con 0x7fa0240ad9f0 2024-03-29T11:19:29.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0240ad5d0 con 0x7fa0240ad9f0 2024-03-29T11:19:29.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa01c001be0 con 0x7fa0240ad9f0 2024-03-29T11:19:29.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:29.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient(hunting): dump: 2024-03-29T11:19:29.111 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:29.111 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:29.111 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.111 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.111 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.111 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:29.111 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3870163693 0 0) 0x7fa024133c20 con 0x7fa0240b8240 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 >> v1:172.21.15.67:6790/0 conn(0x7fa0240ad9f0 legacy=0x7fa0241e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 >> v1:172.21.15.67:6789/0 conn(0x7fa02409c8c0 legacy=0x7fa0241347c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 monclient: found mon.b 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0241351b0 con 0x7fa0240b8240 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient: _finish_auth 0 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:29.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa0298f1640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa0298f1640 10 monclient: _renew_subs 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa0298f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa0298f1640 1 -- 172.21.15.82:0/309530948 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0241356d0 con 0x7fa0240b8240 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa0298f1640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa0298f1640 10 monclient: _renew_subs 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa0298f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa0298f1640 1 -- 172.21.15.82:0/309530948 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02413be70 con 0x7fa0240b8240 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa0298f1640 1 librados: init done 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa00c001170 con 0x7fa0240b8240 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa00c004d90 con 0x7fa0240b8240 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa0217fa640 4 set_mon_vals no callback set 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 10 monclient: dump: 2024-03-29T11:19:29.113 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.119+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa00c0052c0 con 0x7fa0240b8240 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.120+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2218470164 0 0) 0x7fa00c005bd0 con 0x7fa0240b8240 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.120+0000 7fa01bfff640 10 client.4153.objecter ms_dispatch 0x7fa024000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.120+0000 7fa01bfff640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.120+0000 7fa01bfff640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:19:29.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.120+0000 7fa01bfff640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T11:19:29.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.120+0000 7fa0298f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:29.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.120+0000 7fa0298f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:29.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.120+0000 7fa0298f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:29.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.120+0000 7fa0298f1640 1 -- 172.21.15.82:0/309530948 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa008005080 con 0x7fa0240b8240 2024-03-29T11:19:29.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.122+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa00c005e40 con 0x7fa0240b8240 2024-03-29T11:19:29.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.122+0000 7fa01bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:29.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.122+0000 7fa01bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:29.172 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.195+0000 7fa0298f1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e857aa17-61f7-45e4-824c-d5075c3ebf38", "id": 7}] 2024-03-29T11:19:29.172 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.195+0000 7fa0298f1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e857aa17-61f7-45e4-824c-d5075c3ebf38", "id": 7}] 2024-03-29T11:19:29.172 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.195+0000 7fa0298f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:29.172 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.195+0000 7fa0298f1640 1 -- 172.21.15.82:0/309530948 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e857aa17-61f7-45e4-824c-d5075c3ebf38", "id": 7} v 0) v1 -- 0x7fa0080058b0 con 0x7fa0240b8240 2024-03-29T11:19:29.175 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.199+0000 7fa01bfff640 1 -- 172.21.15.82:0/309530948 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e857aa17-61f7-45e4-824c-d5075c3ebf38", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3620998100 0 1371665406) 0x7fa00c005570 con 0x7fa0240b8240 2024-03-29T11:19:29.176 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.199+0000 7fa01bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e857aa17-61f7-45e4-824c-d5075c3ebf38", "id": 7}] 2024-03-29T11:19:29.176 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.199+0000 7fa01bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:29.176 INFO:teuthology.orchestra.run.smithi082.stdout:7 2024-03-29T11:19:29.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.200+0000 7fa0298f1640 10 librados: watch_flush enter 2024-03-29T11:19:29.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.200+0000 7fa0298f1640 10 librados: watch_flush exit 2024-03-29T11:19:29.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.200+0000 7fa0298f1640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T11:19:29.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.200+0000 7fa0298f1640 10 client.4153.objecter successfully canceled tick 2024-03-29T11:19:29.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.200+0000 7fa0298f1640 10 monclient: shutdown 2024-03-29T11:19:29.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.200+0000 7fa0298f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:29.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.200+0000 7fa0298f1640 1 -- 172.21.15.82:0/309530948 >> v1:172.21.15.82:6789/0 conn(0x7fa0240b8240 legacy=0x7fa024131e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.200+0000 7fa0298f1640 1 -- 172.21.15.82:0/309530948 shutdown_connections 2024-03-29T11:19:29.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.200+0000 7fa0298f1640 1 -- 172.21.15.82:0/309530948 >> 172.21.15.82:0/309530948 conn(0x7fa024066b00 msgr2=0x7fa02406bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:29.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.200+0000 7fa0298f1640 1 -- 172.21.15.82:0/309530948 shutdown_connections 2024-03-29T11:19:29.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.200+0000 7fa0298f1640 1 -- 172.21.15.82:0/309530948 wait complete. 2024-03-29T11:19:29.178 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T11:19:29.200+0000 7fa0298f1640 1 librados: shutdown 2024-03-29T11:19:29.187 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:19:29.187 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:19:29.188 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:29.211+0000 7f0c9364b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:29.189 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:19:29.189 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:19:29.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:19:29.190 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:29.213+0000 7fcb6aa38040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:29.192 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:19:29.192 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:19:29.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:19:29.194 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:19:29.194 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:19:29.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:19:29.196 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:19:29.196 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:19:29.196 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:19:29.198 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:19:29.198 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:19:29.199 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:19:29.201 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:19:29.201 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:19:29.201 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:19:29.202 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:19:29.203 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:19:29.203 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:19:29.205 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:19:29.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:29.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.351+0000 7f43d9be6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:29.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.351+0000 7f43d9be6640 10 monclient: build_initial_monmap 2024-03-29T11:19:29.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.351+0000 7f43d9be6640 10 monclient: monmap: 2024-03-29T11:19:29.328 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-03-29T11:19:29.328 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:29.328 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-03-29T11:19:29.352304+0000 2024-03-29T11:19:29.328 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-03-29T11:19:29.352304+0000 2024-03-29T11:19:29.328 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:29.328 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:29.328 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:29.329 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:29.329 INFO:teuthology.orchestra.run.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:29.329 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:19:29.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.352+0000 7f43d9be6640 1 Processor -- start 2024-03-29T11:19:29.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.352+0000 7f43d9be6640 1 -- start start 2024-03-29T11:19:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.352+0000 7f43d9be6640 10 monclient: init 2024-03-29T11:19:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.352+0000 7f43d9be6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.352+0000 7f43d9be6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.352+0000 7f43d9be6640 10 monclient(hunting): picked mon.noname-b con 0x7f43d40b0ad0 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.352+0000 7f43d9be6640 10 monclient(hunting): picked mon.noname-a con 0x7f43d4142320 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.352+0000 7f43d9be6640 10 monclient(hunting): picked mon.noname-c con 0x7f43d406f430 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.352+0000 7f43d9be6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d4142780 con 0x7f43d4142320 2024-03-29T11:19:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.352+0000 7f43d9be6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d4072690 con 0x7f43d406f430 2024-03-29T11:19:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.352+0000 7f43d9be6640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d4071230 con 0x7f43d40b0ad0 2024-03-29T11:19:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.352+0000 7f43d9be6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.352+0000 7f43d9be6640 10 monclient(hunting): authenticate will time out at 829.648871s 2024-03-29T11:19:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d37fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f43d40b0ad0 0x7f43d4071f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:35638/0 (socket says 172.21.15.67:35638) 2024-03-29T11:19:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d2ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f43d4142320 0x7f43d406ed20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43958/0 (socket says 172.21.15.67:43958) 2024-03-29T11:19:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d37fe640 1 -- 172.21.15.67:0/1071273481 learned_addr learned my addr 172.21.15.67:0/1071273481 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:19:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 1 -- 172.21.15.67:0/1071273481 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509889694 0 0) 0x7f43d4142780 con 0x7f43d4142320 2024-03-29T11:19:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:19:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 1 -- 172.21.15.67:0/1071273481 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43bc003520 con 0x7f43d4142320 2024-03-29T11:19:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 1 -- 172.21.15.67:0/1071273481 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310880566 0 0) 0x7f43d4071230 con 0x7f43d40b0ad0 2024-03-29T11:19:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:19:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 1 -- 172.21.15.67:0/1071273481 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d4142780 con 0x7f43d40b0ad0 2024-03-29T11:19:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 1 -- 172.21.15.67:0/1071273481 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415049647 0 0) 0x7f43d4072690 con 0x7f43d406f430 2024-03-29T11:19:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:19:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 1 -- 172.21.15.67:0/1071273481 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d4071230 con 0x7f43d406f430 2024-03-29T11:19:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 1 -- 172.21.15.67:0/1071273481 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1628412141 0 0) 0x7f43bc003520 con 0x7f43d4142320 2024-03-29T11:19:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 1 -- 172.21.15.67:0/1071273481 >> v1:172.21.15.67:6790/0 conn(0x7f43d40b0ad0 legacy=0x7f43d4071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 1 -- 172.21.15.67:0/1071273481 >> v1:172.21.15.82:6789/0 conn(0x7f43d406f430 legacy=0x7f43d414a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 1 monclient: found mon.noname-a 2024-03-29T11:19:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 1 -- 172.21.15.67:0/1071273481 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d40714a0 con 0x7f43d4142320 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 1 -- 172.21.15.67:0/1071273481 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f43c8001690 con 0x7f43d4142320 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient: dump: 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.340 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d9be6640 5 monclient: authenticate success, global_id 4188 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d9be6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 1 -- 172.21.15.67:0/1071273481 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43c80017f0 con 0x7f43d4142320 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 1 -- 172.21.15.67:0/1071273481 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f43c8001c00 con 0x7f43d4142320 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d9be6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d9be6640 4 set_mon_vals no callback set 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.353+0000 7f43d27fc640 10 monclient: dump: 2024-03-29T11:19:29.341 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-03-29T11:19:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:29.342 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:19:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.354+0000 7f43d9be6640 10 monclient: shutdown 2024-03-29T11:19:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.354+0000 7f43d9be6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.354+0000 7f43d9be6640 1 -- 172.21.15.67:0/1071273481 >> v1:172.21.15.67:6789/0 conn(0x7f43d4142320 legacy=0x7f43d406ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.354+0000 7f43d9be6640 1 -- 172.21.15.67:0/1071273481 shutdown_connections 2024-03-29T11:19:29.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.354+0000 7f43d9be6640 1 -- 172.21.15.67:0/1071273481 >> 172.21.15.67:0/1071273481 conn(0x7f43d40b4970 msgr2=0x7f43d40b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:29.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.354+0000 7f43d9be6640 1 -- 172.21.15.67:0/1071273481 shutdown_connections 2024-03-29T11:19:29.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.354+0000 7f43d9be6640 1 -- 172.21.15.67:0/1071273481 wait complete. 2024-03-29T11:19:29.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.355+0000 7f43d9be6640 10 monclient: build_initial_monmap 2024-03-29T11:19:29.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.355+0000 7f43d9be6640 10 monclient: monmap: 2024-03-29T11:19:29.343 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2024-03-29T11:19:29.343 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:29.343 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2024-03-29T11:19:29.343 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2024-03-29T11:19:29.343 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:29.343 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:29.343 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:29.343 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:29.343 INFO:teuthology.orchestra.run.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:29.343 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:19:29.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.355+0000 7f43d9be6640 1 librados: starting msgr at 2024-03-29T11:19:29.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.355+0000 7f43d9be6640 1 librados: starting objecter 2024-03-29T11:19:29.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.355+0000 7f43d9be6640 1 Processor -- start 2024-03-29T11:19:29.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.355+0000 7f43d9be6640 1 -- start start 2024-03-29T11:19:29.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.355+0000 7f43d9be6640 1 librados: setting wanted keys 2024-03-29T11:19:29.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.355+0000 7f43d9be6640 1 librados: calling monclient init 2024-03-29T11:19:29.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.355+0000 7f43d9be6640 10 monclient: init 2024-03-29T11:19:29.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43d9be6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:29.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43d9be6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:29.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43d9be6640 10 monclient(hunting): picked mon.noname-a con 0x7f43d406f430 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:29.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43d9be6640 10 monclient(hunting): picked mon.noname-b con 0x7f43d40b0ad0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:29.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43d37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f43d406f430 0x7f43d41405a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43972/0 (socket says 172.21.15.67:43972) 2024-03-29T11:19:29.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43d9be6640 10 monclient(hunting): picked mon.noname-c con 0x7f43d4142320 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:29.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43d37fe640 1 -- 172.21.15.67:0/3119264494 learned_addr learned my addr 172.21.15.67:0/3119264494 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:19:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43d9be6640 1 -- 172.21.15.67:0/3119264494 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d41d6f70 con 0x7f43d406f430 2024-03-29T11:19:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 client.?.objecter ms_handle_connect 0x7f43d406f430 2024-03-29T11:19:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43d9be6640 1 -- 172.21.15.67:0/3119264494 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d413f920 con 0x7f43d40b0ad0 2024-03-29T11:19:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 client.?.objecter ms_handle_connect 0x7f43d4142320 2024-03-29T11:19:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43d9be6640 1 -- 172.21.15.67:0/3119264494 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d413fc00 con 0x7f43d4142320 2024-03-29T11:19:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43d9be6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43d9be6640 10 monclient(hunting): authenticate will time out at 829.652699s 2024-03-29T11:19:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 client.?.objecter ms_handle_connect 0x7f43d40b0ad0 2024-03-29T11:19:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845726585 0 0) 0x7f43d41d6f70 con 0x7f43d406f430 2024-03-29T11:19:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:19:29.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43c0004eb0 con 0x7f43d406f430 2024-03-29T11:19:29.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073131446 0 0) 0x7f43d413fc00 con 0x7f43d4142320 2024-03-29T11:19:29.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:19:29.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d41d6f70 con 0x7f43d4142320 2024-03-29T11:19:29.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161122912 0 0) 0x7f43d413f920 con 0x7f43d40b0ad0 2024-03-29T11:19:29.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:19:29.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.356+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d413fc00 con 0x7f43d40b0ad0 2024-03-29T11:19:29.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2566733465 0 0) 0x7f43d41d6f70 con 0x7f43d4142320 2024-03-29T11:19:29.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43d413f920 con 0x7f43d4142320 2024-03-29T11:19:29.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2676980883 0 0) 0x7f43c0004eb0 con 0x7f43d406f430 2024-03-29T11:19:29.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43d41d6f70 con 0x7f43d406f430 2024-03-29T11:19:29.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f43cc001bf0 con 0x7f43d4142320 2024-03-29T11:19:29.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:29.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient(hunting): dump: 2024-03-29T11:19:29.347 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-03-29T11:19:29.347 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:29.347 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.347 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f43c4001970 con 0x7f43d406f430 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient(hunting): dump: 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.348 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.349 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.349 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:29.349 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:29.349 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:29.349 INFO:teuthology.orchestra.run.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:29.349 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:19:29.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1947474957 0 0) 0x7f43d413fc00 con 0x7f43d40b0ad0 2024-03-29T11:19:29.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43c0004eb0 con 0x7f43d40b0ad0 2024-03-29T11:19:29.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f43c8001b00 con 0x7f43d40b0ad0 2024-03-29T11:19:29.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:29.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient(hunting): dump: 2024-03-29T11:19:29.349 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-03-29T11:19:29.349 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:29.350 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.350 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.350 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.350 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:29.350 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:29.350 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:29.350 INFO:teuthology.orchestra.run.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:29.350 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:19:29.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4127666707 0 0) 0x7f43d413f920 con 0x7f43d4142320 2024-03-29T11:19:29.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 >> v1:172.21.15.82:6789/0 conn(0x7f43d40b0ad0 legacy=0x7f43d413d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 >> v1:172.21.15.67:6789/0 conn(0x7f43d406f430 legacy=0x7f43d41405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:29.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 monclient: found mon.c 2024-03-29T11:19:29.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d41d7400 con 0x7f43d4142320 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient: _finish_auth 0 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43d9be6640 5 monclient: authenticate success, global_id 4184 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43d9be6640 10 monclient: _renew_subs 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43d9be6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43d9be6640 1 -- 172.21.15.67:0/3119264494 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d4140cb0 con 0x7f43d4142320 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43d9be6640 10 client.4184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43d9be6640 10 monclient: _renew_subs 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43d9be6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43d9be6640 1 -- 172.21.15.67:0/3119264494 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d41d6a50 con 0x7f43d4142320 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43d9be6640 1 librados: init done 2024-03-29T11:19:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43cc003990 con 0x7f43d4142320 2024-03-29T11:19:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f43cc003ca0 con 0x7f43d4142320 2024-03-29T11:19:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43d1ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 monclient: dump: 2024-03-29T11:19:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2024-03-29T11:19:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:29.352 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:29.353 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:19:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f43cc004190 con 0x7f43d4142320 2024-03-29T11:19:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f43cc004b10 con 0x7f43d4142320 2024-03-29T11:19:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 10 client.4184.objecter ms_dispatch 0x7f43d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 3 client.4184.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 3 client.4184.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.357+0000 7f43bbfff640 20 client.4184.objecter dump_active .. 0 homeless 2024-03-29T11:19:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.358+0000 7f43d9be6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.358+0000 7f43d9be6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.358+0000 7f43d9be6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:19:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.358+0000 7f43d9be6640 1 -- 172.21.15.67:0/3119264494 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b4005080 con 0x7f43d4142320 2024-03-29T11:19:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.360+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f43cc004da0 con 0x7f43d4142320 2024-03-29T11:19:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.360+0000 7f43bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:29.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.360+0000 7f43bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:29.354 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:29.369+0000 7f0c9364b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:29.354 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:29.362+0000 7fcb6aa38040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:29.385 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:29.408+0000 7fcb6aa38040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:29.394 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:29.417+0000 7f0c9364b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:29.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.421+0000 7f43d9be6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:29.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.421+0000 7f43d9be6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:29.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.422+0000 7f43d9be6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:19:29.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.422+0000 7f43d9be6640 1 -- 172.21.15.67:0/3119264494 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43b4005330 con 0x7f43d4142320 2024-03-29T11:19:29.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.422+0000 7f43bbfff640 1 -- 172.21.15.67:0/3119264494 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f43cc004440 con 0x7f43d4142320 2024-03-29T11:19:29.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.422+0000 7f43bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:29.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.422+0000 7f43bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:29.399 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-29T11:19:29.399 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:29.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.423+0000 7f43d9be6640 10 librados: watch_flush enter 2024-03-29T11:19:29.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.423+0000 7f43d9be6640 10 librados: watch_flush exit 2024-03-29T11:19:29.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.423+0000 7f43d9be6640 20 client.4184.objecter shutdown clearing up homeless session... 2024-03-29T11:19:29.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.423+0000 7f43d9be6640 10 client.4184.objecter successfully canceled tick 2024-03-29T11:19:29.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.423+0000 7f43d9be6640 10 monclient: shutdown 2024-03-29T11:19:29.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.423+0000 7f43d9be6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:29.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.423+0000 7f43d9be6640 1 -- 172.21.15.67:0/3119264494 >> v1:172.21.15.67:6790/0 conn(0x7f43d4142320 legacy=0x7f43d41ddbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.423+0000 7f43d9be6640 1 -- 172.21.15.67:0/3119264494 shutdown_connections 2024-03-29T11:19:29.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.423+0000 7f43d9be6640 1 -- 172.21.15.67:0/3119264494 >> 172.21.15.67:0/3119264494 conn(0x7f43d40b4970 msgr2=0x7f43d406ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:29.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.423+0000 7f43d9be6640 1 -- 172.21.15.67:0/3119264494 shutdown_connections 2024-03-29T11:19:29.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.423+0000 7f43d9be6640 1 -- 172.21.15.67:0/3119264494 wait complete. 2024-03-29T11:19:29.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-29T11:19:29.423+0000 7f43d9be6640 1 librados: shutdown 2024-03-29T11:19:29.410 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:19:29.410 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:19:29.496 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:29.520+0000 7fcb6aa38040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:29.512 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:29.536+0000 7f0c9364b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:29.592 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:29.615+0000 7fcb6aa38040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:29.612 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:29.635+0000 7f0c9364b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:29.744 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:29.767+0000 7fcb6aa38040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:29.773 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:29.797+0000 7f0c9364b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:29.785 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:29.809+0000 7fcb6aa38040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:29.792 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-29T11:19:29.815+0000 7f9eb2593640 -1 Falling back to public interface 2024-03-29T11:19:29.799 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-29T11:19:29.822+0000 7ff694e99640 -1 Falling back to public interface 2024-03-29T11:19:29.800 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-29T11:19:29.823+0000 7f684b812640 -1 Falling back to public interface 2024-03-29T11:19:29.804 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-29T11:19:29.827+0000 7f0015037640 -1 Falling back to public interface 2024-03-29T11:19:29.818 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:29.841+0000 7f0c9364b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:29.820 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-29T11:19:29.844+0000 7f2dc3106640 -1 Falling back to public interface 2024-03-29T11:19:29.833 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-29T11:19:29.856+0000 7f1d0cc89640 -1 Falling back to public interface 2024-03-29T11:19:29.833 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-29T11:19:29.856+0000 7f52994d9640 -1 Falling back to public interface 2024-03-29T11:19:29.834 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-29T11:19:29.857+0000 7ffa2e79e640 -1 Falling back to public interface 2024-03-29T11:19:29.943 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:29.966+0000 7fcb6aa38040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:29.982 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:30.005+0000 7f0c9364b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:30.216 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:30.306 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.329+0000 7f7b04c3e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:30.306 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.329+0000 7f7b04c3e640 10 monclient: build_initial_monmap 2024-03-29T11:19:30.306 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.329+0000 7f7b04c3e640 10 monclient: monmap: 2024-03-29T11:19:30.306 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:19:30.306 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:30.306 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:19:30.329933+0000 2024-03-29T11:19:30.306 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:19:30.329933+0000 2024-03-29T11:19:30.307 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:30.307 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:30.307 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:30.307 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:30.307 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:30.307 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:30.307 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.329+0000 7f7b04c3e640 1 Processor -- start 2024-03-29T11:19:30.307 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.329+0000 7f7b04c3e640 1 -- start start 2024-03-29T11:19:30.307 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.329+0000 7f7b04c3e640 10 monclient: init 2024-03-29T11:19:30.315 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7b04c3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:30.315 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7b04c3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:30.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7b04c3e640 10 monclient(hunting): picked mon.noname-b con 0x7f7b001457f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:30.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7b04c3e640 10 monclient(hunting): picked mon.noname-c con 0x7f7b0013f650 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:30.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7b04c3e640 10 monclient(hunting): picked mon.noname-a con 0x7f7b000a9d70 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:30.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7b04c3e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b000aaef0 con 0x7f7b000a9d70 2024-03-29T11:19:30.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7b04c3e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b00145bd0 con 0x7f7b0013f650 2024-03-29T11:19:30.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7b04c3e640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b00145eb0 con 0x7f7b001457f0 2024-03-29T11:19:30.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7b04c3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:30.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7b04c3e640 10 monclient(hunting): authenticate will time out at 830.626403s 2024-03-29T11:19:30.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7b000a9d70 0x7f7b0014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43988/0 (socket says 172.21.15.67:43988) 2024-03-29T11:19:30.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afe59c640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f7b001457f0 0x7f7b0013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:35646/0 (socket says 172.21.15.67:35646) 2024-03-29T11:19:30.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afed9d640 1 -- 172.21.15.67:0/2876712451 learned_addr learned my addr 172.21.15.67:0/2876712451 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:19:30.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afd59a640 1 -- 172.21.15.67:0/2876712451 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990312425 0 0) 0x7f7b00145eb0 con 0x7f7b001457f0 2024-03-29T11:19:30.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:30.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:30.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:30.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afd59a640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:19:30.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afd59a640 1 -- 172.21.15.67:0/2876712451 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ae8003520 con 0x7f7b001457f0 2024-03-29T11:19:30.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afd59a640 1 -- 172.21.15.67:0/2876712451 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186204314 0 0) 0x7f7b000aaef0 con 0x7f7b000a9d70 2024-03-29T11:19:30.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:30.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:30.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:30.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afd59a640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:19:30.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.330+0000 7f7afd59a640 1 -- 172.21.15.67:0/2876712451 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b00145eb0 con 0x7f7b000a9d70 2024-03-29T11:19:30.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 1 -- 172.21.15.67:0/2876712451 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444939285 0 0) 0x7f7b00145bd0 con 0x7f7b0013f650 2024-03-29T11:19:30.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:30.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:30.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:30.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:19:30.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 1 -- 172.21.15.67:0/2876712451 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b000aaef0 con 0x7f7b0013f650 2024-03-29T11:19:30.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 1 -- 172.21.15.67:0/2876712451 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 36913616 0 0) 0x7f7b00145eb0 con 0x7f7b000a9d70 2024-03-29T11:19:30.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 1 -- 172.21.15.67:0/2876712451 >> v1:172.21.15.67:6790/0 conn(0x7f7b001457f0 legacy=0x7f7b0013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:30.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 1 -- 172.21.15.67:0/2876712451 >> v1:172.21.15.82:6789/0 conn(0x7f7b0013f650 legacy=0x7f7b000b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:30.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:19:30.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 1 monclient: found mon.noname-a 2024-03-29T11:19:30.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:30.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:19:30.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 1 -- 172.21.15.67:0/2876712451 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b00146340 con 0x7f7b000a9d70 2024-03-29T11:19:30.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient: _finish_auth 0 2024-03-29T11:19:30.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:19:30.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:30.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 1 -- 172.21.15.67:0/2876712451 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7af8001970 con 0x7f7b000a9d70 2024-03-29T11:19:30.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:30.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:30.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient: dump: 2024-03-29T11:19:30.319 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:30.319 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:30.319 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:30.319 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:30.319 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:30.319 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:30.320 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:30.320 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:30.320 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:30.320 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:30.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7b04c3e640 5 monclient: authenticate success, global_id 4194 2024-03-29T11:19:30.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7b04c3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:30.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 1 -- 172.21.15.67:0/2876712451 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7af8001ad0 con 0x7f7b000a9d70 2024-03-29T11:19:30.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:30.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 1 -- 172.21.15.67:0/2876712451 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7af8001f20 con 0x7f7b000a9d70 2024-03-29T11:19:30.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7b04c3e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:30.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7b04c3e640 4 set_mon_vals no callback set 2024-03-29T11:19:30.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:30.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:30.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7afd59a640 10 monclient: dump: 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7b04c3e640 10 monclient: shutdown 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7b04c3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.331+0000 7f7b04c3e640 1 -- 172.21.15.67:0/2876712451 >> v1:172.21.15.67:6789/0 conn(0x7f7b000a9d70 legacy=0x7f7b0014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.332+0000 7f7b04c3e640 1 -- 172.21.15.67:0/2876712451 shutdown_connections 2024-03-29T11:19:30.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.332+0000 7f7b04c3e640 1 -- 172.21.15.67:0/2876712451 >> 172.21.15.67:0/2876712451 conn(0x7f7b00066a00 msgr2=0x7f7b00066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.332+0000 7f7b04c3e640 1 -- 172.21.15.67:0/2876712451 shutdown_connections 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.332+0000 7f7b04c3e640 1 -- 172.21.15.67:0/2876712451 wait complete. 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.332+0000 7f7b04c3e640 10 monclient: build_initial_monmap 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.332+0000 7f7b04c3e640 10 monclient: monmap: 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:30.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 1 librados: starting msgr at 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 1 librados: starting objecter 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 1 Processor -- start 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 1 -- start start 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 1 librados: setting wanted keys 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 1 librados: calling monclient init 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 10 monclient: init 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 10 monclient(hunting): picked mon.noname-c con 0x7f7b000a9d70 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 10 monclient(hunting): picked mon.noname-b con 0x7f7b0013f650 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 10 monclient(hunting): picked mon.noname-a con 0x7f7b001457f0 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b00142400 con 0x7f7b001457f0 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b001420e0 con 0x7f7b0013f650 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b000ac360 con 0x7f7b000a9d70 2024-03-29T11:19:30.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:30.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7b04c3e640 10 monclient(hunting): authenticate will time out at 830.629640s 2024-03-29T11:19:30.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7afe59c640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f7b000a9d70 0x7f7b001d7470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:35648/0 (socket says 172.21.15.67:35648) 2024-03-29T11:19:30.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7afe59c640 1 -- 172.21.15.67:0/667080546 learned_addr learned my addr 172.21.15.67:0/667080546 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:19:30.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7afed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7b001457f0 0x7f7b001419d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43992/0 (socket says 172.21.15.67:43992) 2024-03-29T11:19:30.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7aeeffd640 10 client.?.objecter ms_handle_connect 0x7f7b001457f0 2024-03-29T11:19:30.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7aeeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:30.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7aeeffd640 10 client.?.objecter ms_handle_connect 0x7f7b000a9d70 2024-03-29T11:19:30.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.333+0000 7f7aeeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:30.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302801878 0 0) 0x7f7b00142400 con 0x7f7b001457f0 2024-03-29T11:19:30.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:30.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:30.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:30.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): my global_id is 4197 2024-03-29T11:19:30.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ae0004eb0 con 0x7f7b001457f0 2024-03-29T11:19:30.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 client.?.objecter ms_handle_connect 0x7f7b0013f650 2024-03-29T11:19:30.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:30.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66203569 0 0) 0x7f7b000ac360 con 0x7f7b000a9d70 2024-03-29T11:19:30.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:30.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:30.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:30.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:19:30.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b00142400 con 0x7f7b000a9d70 2024-03-29T11:19:30.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963043723 0 0) 0x7f7b001420e0 con 0x7f7b0013f650 2024-03-29T11:19:30.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:30.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:30.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:30.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:19:30.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b000ac360 con 0x7f7b0013f650 2024-03-29T11:19:30.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1938766562 0 0) 0x7f7ae0004eb0 con 0x7f7b001457f0 2024-03-29T11:19:30.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b001420e0 con 0x7f7b001457f0 2024-03-29T11:19:30.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7af8001930 con 0x7f7b001457f0 2024-03-29T11:19:30.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:30.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:30.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): dump: 2024-03-29T11:19:30.326 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:30.326 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:30.326 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:30.326 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:30.326 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:30.326 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:30.326 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:30.326 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:30.327 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:30.327 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:30.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:30.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 673308495 0 0) 0x7f7b00142400 con 0x7f7b000a9d70 2024-03-29T11:19:30.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ae0004eb0 con 0x7f7b000a9d70 2024-03-29T11:19:30.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7af0001850 con 0x7f7b000a9d70 2024-03-29T11:19:30.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:30.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:30.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): dump: 2024-03-29T11:19:30.327 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:30.327 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:30.327 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:30.327 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:30.327 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:30.328 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:30.328 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:30.328 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:30.328 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:30.328 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:30.328 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3609007482 0 0) 0x7f7b000ac360 con 0x7f7b0013f650 2024-03-29T11:19:30.328 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b00142400 con 0x7f7b0013f650 2024-03-29T11:19:30.328 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7af4001830 con 0x7f7b0013f650 2024-03-29T11:19:30.328 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:30.328 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:30.328 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient(hunting): dump: 2024-03-29T11:19:30.328 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:30.328 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:30.329 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:30.329 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:30.329 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:30.329 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:30.329 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:30.329 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:30.329 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:30.329 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:30.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3157493315 0 0) 0x7f7b001420e0 con 0x7f7b001457f0 2024-03-29T11:19:30.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 >> v1:172.21.15.67:6790/0 conn(0x7f7b000a9d70 legacy=0x7f7b001d7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:30.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 >> v1:172.21.15.82:6789/0 conn(0x7f7b0013f650 legacy=0x7f7b00142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:30.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:30.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 monclient: found mon.a 2024-03-29T11:19:30.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:30.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:30.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b00144890 con 0x7f7b001457f0 2024-03-29T11:19:30.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:30.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:30.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:30.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7b04c3e640 5 monclient: authenticate success, global_id 4197 2024-03-29T11:19:30.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7b04c3e640 10 monclient: _renew_subs 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7b04c3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7b04c3e640 1 -- 172.21.15.67:0/667080546 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b00144b40 con 0x7f7b001457f0 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7b04c3e640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7b04c3e640 10 monclient: _renew_subs 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7b04c3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7b04c3e640 1 -- 172.21.15.67:0/667080546 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b00137160 con 0x7f7b001457f0 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7b04c3e640 1 librados: init done 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7af8004e10 con 0x7f7b001457f0 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7af8001fa0 con 0x7f7b001457f0 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 monclient: dump: 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:30.331 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7afcd99640 4 set_mon_vals no callback set 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7af80054f0 con 0x7f7b001457f0 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f7af8006df0 con 0x7f7b001457f0 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 10 client.4197.objecter ms_dispatch 0x7f7b00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 3 client.4197.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 3 client.4197.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.334+0000 7f7aeeffd640 20 client.4197.objecter dump_active .. 0 homeless 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.335+0000 7f7b04c3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:30.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.335+0000 7f7b04c3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:30.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.335+0000 7f7b04c3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:30.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.335+0000 7f7b04c3e640 1 -- 172.21.15.67:0/667080546 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae4005080 con 0x7f7b001457f0 2024-03-29T11:19:30.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.337+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7af8006720 con 0x7f7b001457f0 2024-03-29T11:19:30.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.337+0000 7f7aeeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:30.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.337+0000 7f7aeeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:30.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.399+0000 7f7b04c3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:30.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.400+0000 7f7b04c3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:30.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.400+0000 7f7b04c3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:30.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.400+0000 7f7b04c3e640 1 -- 172.21.15.67:0/667080546 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ae4005330 con 0x7f7b001457f0 2024-03-29T11:19:30.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.400+0000 7f7aeeffd640 1 -- 172.21.15.67:0/667080546 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f7af80069d0 con 0x7f7b001457f0 2024-03-29T11:19:30.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.400+0000 7f7aeeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:30.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.400+0000 7f7aeeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:30.378 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:19:30.378 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:30.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.401+0000 7f7b04c3e640 10 librados: watch_flush enter 2024-03-29T11:19:30.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.401+0000 7f7b04c3e640 10 librados: watch_flush exit 2024-03-29T11:19:30.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.401+0000 7f7b04c3e640 20 client.4197.objecter shutdown clearing up homeless session... 2024-03-29T11:19:30.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.401+0000 7f7b04c3e640 10 client.4197.objecter successfully canceled tick 2024-03-29T11:19:30.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.401+0000 7f7b04c3e640 10 monclient: shutdown 2024-03-29T11:19:30.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.401+0000 7f7b04c3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:30.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.401+0000 7f7b04c3e640 1 -- 172.21.15.67:0/667080546 >> v1:172.21.15.67:6789/0 conn(0x7f7b001457f0 legacy=0x7f7b001419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:30.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.401+0000 7f7b04c3e640 1 -- 172.21.15.67:0/667080546 shutdown_connections 2024-03-29T11:19:30.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.401+0000 7f7b04c3e640 1 -- 172.21.15.67:0/667080546 >> 172.21.15.67:0/667080546 conn(0x7f7b00066a00 msgr2=0x7f7b000b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:30.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.401+0000 7f7b04c3e640 1 -- 172.21.15.67:0/667080546 shutdown_connections 2024-03-29T11:19:30.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.401+0000 7f7b04c3e640 1 -- 172.21.15.67:0/667080546 wait complete. 2024-03-29T11:19:30.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:30.401+0000 7f7b04c3e640 1 librados: shutdown 2024-03-29T11:19:30.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:30.951 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:30.974+0000 7f0c9364b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:31.017 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:31.040+0000 7fcb6aa38040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:31.239 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:31.262+0000 7f0c9364b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:19:31.288 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:31.311+0000 7fcb6aa38040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:19:31.292 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:31.316+0000 7f0c9364b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:19:31.338 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:31.361+0000 7fcb6aa38040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:19:31.413 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:31.436+0000 7f0c9364b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:19:31.459 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T11:19:31.483+0000 7fcb6aa38040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:19:32.426 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:32.449+0000 7f0c3429d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:19:32.430 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-29T11:19:32.450+0000 7f0c3429d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:19:33.271 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-29T11:19:33.295+0000 7f9eb2593640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:19:33.274 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-29T11:19:33.297+0000 7f684b812640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:19:33.321 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-29T11:19:33.344+0000 7f2dc3106640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:19:33.323 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-29T11:19:33.347+0000 7ffa2e79e640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:19:33.349 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-29T11:19:33.373+0000 7f1d0cc89640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:19:33.354 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-29T11:19:33.378+0000 7ff694e99640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:19:33.360 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-29T11:19:33.383+0000 7f0015037640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:19:33.389 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-29T11:19:33.413+0000 7f52994d9640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:19:37.193 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:37.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.373+0000 7fa8eebac640 10 monclient: get_monmap_and_config 2024-03-29T11:19:37.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.373+0000 7fa8eebac640 10 monclient: build_initial_monmap 2024-03-29T11:19:37.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.373+0000 7fa8eebac640 10 monclient: monmap: 2024-03-29T11:19:37.350 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:19:37.350 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:37.350 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:19:37.374128+0000 2024-03-29T11:19:37.350 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:19:37.374128+0000 2024-03-29T11:19:37.350 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:37.350 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:37.350 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:37.351 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:37.351 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:37.351 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:37.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.374+0000 7fa8eebac640 1 Processor -- start 2024-03-29T11:19:37.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.374+0000 7fa8eebac640 1 -- start start 2024-03-29T11:19:37.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.374+0000 7fa8eebac640 10 monclient: init 2024-03-29T11:19:37.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.375+0000 7fa8eebac640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:37.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.375+0000 7fa8eebac640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:37.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.375+0000 7fa8eebac640 10 monclient(hunting): picked mon.noname-c con 0x7fa8e806d9d0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:37.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.375+0000 7fa8eebac640 10 monclient(hunting): picked mon.noname-a con 0x7fa8e813ef30 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:37.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.375+0000 7fa8eebac640 10 monclient(hunting): picked mon.noname-b con 0x7fa8e80b0860 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:37.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.375+0000 7fa8eebac640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e80aed10 con 0x7fa8e813ef30 2024-03-29T11:19:37.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.375+0000 7fa8eebac640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e80ab1b0 con 0x7fa8e806d9d0 2024-03-29T11:19:37.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.375+0000 7fa8eebac640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e80ab490 con 0x7fa8e80b0860 2024-03-29T11:19:37.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.375+0000 7fa8eebac640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:37.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.375+0000 7fa8eebac640 10 monclient(hunting): authenticate will time out at 837.671711s 2024-03-29T11:19:37.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.376+0000 7fa8ec948640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fa8e806d9d0 0x7fa8e813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:50114/0 (socket says 172.21.15.67:50114) 2024-03-29T11:19:37.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.376+0000 7fa8ec948640 1 -- 172.21.15.67:0/3933452568 learned_addr learned my addr 172.21.15.67:0/3933452568 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:19:37.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 1 -- 172.21.15.67:0/3933452568 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347638832 0 0) 0x7fa8e80ab1b0 con 0x7fa8e806d9d0 2024-03-29T11:19:37.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:37.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:37.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:37.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient(hunting): my global_id is 4228 2024-03-29T11:19:37.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 1 -- 172.21.15.67:0/3933452568 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d4003520 con 0x7fa8e806d9d0 2024-03-29T11:19:37.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 1 -- 172.21.15.67:0/3933452568 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418357590 0 0) 0x7fa8e80ab490 con 0x7fa8e80b0860 2024-03-29T11:19:37.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:37.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:37.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:37.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient(hunting): my global_id is 4229 2024-03-29T11:19:37.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 1 -- 172.21.15.67:0/3933452568 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e80ab1b0 con 0x7fa8e80b0860 2024-03-29T11:19:37.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 1 -- 172.21.15.67:0/3933452568 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3899427719 0 0) 0x7fa8d4003520 con 0x7fa8e806d9d0 2024-03-29T11:19:37.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 1 -- 172.21.15.67:0/3933452568 >> v1:172.21.15.67:6790/0 conn(0x7fa8e80b0860 legacy=0x7fa8e814ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:37.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 1 -- 172.21.15.67:0/3933452568 >> v1:172.21.15.67:6789/0 conn(0x7fa8e813ef30 legacy=0x7fa8e80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:37.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:37.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 1 monclient: found mon.noname-c 2024-03-29T11:19:37.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:37.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:19:37.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 1 -- 172.21.15.67:0/3933452568 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e80ab770 con 0x7fa8e806d9d0 2024-03-29T11:19:37.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:37.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:37.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:37.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 1 -- 172.21.15.67:0/3933452568 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa8d0001690 con 0x7fa8e806d9d0 2024-03-29T11:19:37.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:37.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:37.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8e77fe640 10 monclient: dump: 2024-03-29T11:19:37.356 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:37.356 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:37.356 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:37.357 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:37.357 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:37.357 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:37.357 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:37.357 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:37.357 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:37.357 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:37.357 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8eebac640 5 monclient: authenticate success, global_id 4228 2024-03-29T11:19:37.357 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.377+0000 7fa8eebac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:37.358 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.378+0000 7fa8e77fe640 1 -- 172.21.15.67:0/3933452568 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8d00017f0 con 0x7fa8e806d9d0 2024-03-29T11:19:37.358 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.378+0000 7fa8e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:37.358 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.378+0000 7fa8e77fe640 1 -- 172.21.15.67:0/3933452568 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa8d0001c40 con 0x7fa8e806d9d0 2024-03-29T11:19:37.358 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.378+0000 7fa8e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:37.358 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.378+0000 7fa8e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:37.358 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.378+0000 7fa8e77fe640 10 monclient: dump: 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.378+0000 7fa8eebac640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.378+0000 7fa8eebac640 4 set_mon_vals no callback set 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.379+0000 7fa8eebac640 10 monclient: shutdown 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.379+0000 7fa8eebac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:37.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.379+0000 7fa8eebac640 1 -- 172.21.15.67:0/3933452568 >> v1:172.21.15.82:6789/0 conn(0x7fa8e806d9d0 legacy=0x7fa8e813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:37.360 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.380+0000 7fa8eebac640 1 -- 172.21.15.67:0/3933452568 shutdown_connections 2024-03-29T11:19:37.360 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.380+0000 7fa8eebac640 1 -- 172.21.15.67:0/3933452568 >> 172.21.15.67:0/3933452568 conn(0x7fa8e806b660 msgr2=0x7fa8e806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:37.360 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.380+0000 7fa8eebac640 1 -- 172.21.15.67:0/3933452568 shutdown_connections 2024-03-29T11:19:37.360 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.380+0000 7fa8eebac640 1 -- 172.21.15.67:0/3933452568 wait complete. 2024-03-29T11:19:37.360 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.381+0000 7fa8eebac640 10 monclient: build_initial_monmap 2024-03-29T11:19:37.360 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.381+0000 7fa8eebac640 10 monclient: monmap: 2024-03-29T11:19:37.360 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:19:37.360 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:37.360 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:19:37.360 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:19:37.360 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:37.360 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:37.360 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:37.360 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:37.361 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:37.361 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:37.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.382+0000 7fa8eebac640 1 librados: starting msgr at 2024-03-29T11:19:37.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.382+0000 7fa8eebac640 1 librados: starting objecter 2024-03-29T11:19:37.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.382+0000 7fa8eebac640 1 Processor -- start 2024-03-29T11:19:37.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.383+0000 7fa8eebac640 1 -- start start 2024-03-29T11:19:37.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.383+0000 7fa8eebac640 1 librados: setting wanted keys 2024-03-29T11:19:37.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.383+0000 7fa8eebac640 1 librados: calling monclient init 2024-03-29T11:19:37.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.383+0000 7fa8eebac640 10 monclient: init 2024-03-29T11:19:37.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.385+0000 7fa8eebac640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:37.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.385+0000 7fa8eebac640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:37.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.385+0000 7fa8eebac640 10 monclient(hunting): picked mon.noname-b con 0x7fa8e806d9d0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:37.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.385+0000 7fa8eebac640 10 monclient(hunting): picked mon.noname-a con 0x7fa8e80b0860 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:37.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.385+0000 7fa8ec948640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fa8e806d9d0 0x7fa8e80b2460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:50326/0 (socket says 172.21.15.67:50326) 2024-03-29T11:19:37.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.385+0000 7fa8ed149640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fa8e813ef30 0x7fa8e81e20c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:42274/0 (socket says 172.21.15.67:42274) 2024-03-29T11:19:37.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.385+0000 7fa8ec948640 1 -- 172.21.15.67:0/4064833693 learned_addr learned my addr 172.21.15.67:0/4064833693 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:19:37.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.386+0000 7fa8eebac640 10 monclient(hunting): picked mon.noname-c con 0x7fa8e813ef30 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:37.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.386+0000 7fa8e57fa640 10 client.?.objecter ms_handle_connect 0x7fa8e806d9d0 2024-03-29T11:19:37.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.386+0000 7fa8e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:37.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.386+0000 7fa8e57fa640 10 client.?.objecter ms_handle_connect 0x7fa8e813ef30 2024-03-29T11:19:37.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.386+0000 7fa8e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:37.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.386+0000 7fa8eebac640 1 -- 172.21.15.67:0/4064833693 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e80b3eb0 con 0x7fa8e80b0860 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.386+0000 7fa8eebac640 1 -- 172.21.15.67:0/4064833693 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e81d75f0 con 0x7fa8e806d9d0 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.386+0000 7fa8eebac640 1 -- 172.21.15.67:0/4064833693 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e8139c70 con 0x7fa8e813ef30 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.386+0000 7fa8eebac640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.386+0000 7fa8eebac640 10 monclient(hunting): authenticate will time out at 837.682398s 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.386+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385807084 0 0) 0x7fa8e8139c70 con 0x7fa8e813ef30 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.386+0000 7fa8e57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.387+0000 7fa8e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.387+0000 7fa8e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.387+0000 7fa8e57fa640 10 monclient(hunting): my global_id is 4232 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.387+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c8004eb0 con 0x7fa8e813ef30 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.387+0000 7fa8e57fa640 10 client.?.objecter ms_handle_connect 0x7fa8e80b0860 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.387+0000 7fa8e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.387+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476460076 0 0) 0x7fa8e81d75f0 con 0x7fa8e806d9d0 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.387+0000 7fa8e57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:37.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.387+0000 7fa8e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.387+0000 7fa8e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.387+0000 7fa8e57fa640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.387+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e8139c70 con 0x7fa8e806d9d0 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753859245 0 0) 0x7fa8e80b3eb0 con 0x7fa8e80b0860 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e81d75f0 con 0x7fa8e80b0860 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4291822266 0 0) 0x7fa8e8139c70 con 0x7fa8e806d9d0 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8e80b3eb0 con 0x7fa8e806d9d0 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa8d0001b40 con 0x7fa8e806d9d0 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:37.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 10 monclient(hunting): dump: 2024-03-29T11:19:37.369 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:37.369 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:37.369 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:37.369 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:37.369 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:37.369 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:37.369 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:37.369 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:37.369 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:37.369 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:37.369 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:37.369 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 909438870 0 0) 0x7fa8c8004eb0 con 0x7fa8e813ef30 2024-03-29T11:19:37.369 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8e8139c70 con 0x7fa8e813ef30 2024-03-29T11:19:37.369 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa8e0001c30 con 0x7fa8e813ef30 2024-03-29T11:19:37.370 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:37.370 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:37.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.388+0000 7fa8e57fa640 10 monclient(hunting): dump: 2024-03-29T11:19:37.373 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:37.373 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:37.373 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:37.373 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:37.373 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:37.373 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:37.374 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:37.374 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:37.374 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:37.374 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:37.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3174609087 0 0) 0x7fa8e80b3eb0 con 0x7fa8e806d9d0 2024-03-29T11:19:37.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 >> v1:172.21.15.67:6790/0 conn(0x7fa8e813ef30 legacy=0x7fa8e81e20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:37.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 >> v1:172.21.15.67:6789/0 conn(0x7fa8e80b0860 legacy=0x7fa8e81d7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:37.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8e57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:37.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8e57fa640 1 monclient: found mon.b 2024-03-29T11:19:37.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:37.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8e57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:37.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e8139e00 con 0x7fa8e806d9d0 2024-03-29T11:19:37.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8e57fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:37.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8e57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:37.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:37.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8eebac640 5 monclient: authenticate success, global_id 4231 2024-03-29T11:19:37.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8eebac640 10 monclient: _renew_subs 2024-03-29T11:19:37.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8eebac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:37.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.389+0000 7fa8eebac640 1 -- 172.21.15.67:0/4064833693 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e80b2c30 con 0x7fa8e806d9d0 2024-03-29T11:19:37.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.390+0000 7fa8eebac640 10 client.4231.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:37.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.390+0000 7fa8eebac640 10 monclient: _renew_subs 2024-03-29T11:19:37.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.390+0000 7fa8eebac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:37.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.390+0000 7fa8eebac640 1 -- 172.21.15.67:0/4064833693 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e81d84b0 con 0x7fa8e806d9d0 2024-03-29T11:19:37.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.390+0000 7fa8eebac640 1 librados: init done 2024-03-29T11:19:37.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.390+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8d0004bf0 con 0x7fa8e806d9d0 2024-03-29T11:19:37.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.390+0000 7fa8e57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:37.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.390+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa8d0001ca0 con 0x7fa8e806d9d0 2024-03-29T11:19:37.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.390+0000 7fa8e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:37.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.390+0000 7fa8e57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:37.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.391+0000 7fa8e57fa640 10 monclient: dump: 2024-03-29T11:19:37.376 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:37.376 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:37.376 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:37.376 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:37.376 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:37.376 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:37.376 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:37.376 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:37.376 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:37.376 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:37.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.391+0000 7fa8e67fc640 4 set_mon_vals no callback set 2024-03-29T11:19:37.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.392+0000 7fa8eebac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:37.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.392+0000 7fa8eebac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:37.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.392+0000 7fa8eebac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:37.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.392+0000 7fa8eebac640 1 -- 172.21.15.67:0/4064833693 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8cc005080 con 0x7fa8e806d9d0 2024-03-29T11:19:37.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.393+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fa8d001abd0 con 0x7fa8e806d9d0 2024-03-29T11:19:37.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.394+0000 7fa8e7fff640 10 monclient: get_auth_request con 0x7fa8c806dc90 auth_method 0 2024-03-29T11:19:37.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.394+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fa8d0084a80 con 0x7fa8e806d9d0 2024-03-29T11:19:37.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.394+0000 7fa8e57fa640 10 client.4231.objecter ms_dispatch 0x7fa8e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:37.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.394+0000 7fa8e57fa640 3 client.4231.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:37.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.394+0000 7fa8e57fa640 3 client.4231.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:37.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.394+0000 7fa8e57fa640 20 client.4231.objecter dump_active .. 0 homeless 2024-03-29T11:19:37.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.394+0000 7fa8e57fa640 10 client.4231.objecter ms_handle_connect 0x7fa8c806dc90 2024-03-29T11:19:37.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.401+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8d0050360 con 0x7fa8e806d9d0 2024-03-29T11:19:37.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.401+0000 7fa8e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:37.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.401+0000 7fa8e57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:37.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.799+0000 7fa8eebac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:37.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.799+0000 7fa8eebac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:37.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.799+0000 7fa8eebac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:37.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.799+0000 7fa8eebac640 1 -- 172.21.15.67:0/4064833693 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8cc005330 con 0x7fa8e806d9d0 2024-03-29T11:19:37.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.800+0000 7fa8e57fa640 1 -- 172.21.15.67:0/4064833693 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fa8d0052290 con 0x7fa8e806d9d0 2024-03-29T11:19:37.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.800+0000 7fa8e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:37.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.800+0000 7fa8e57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:37.777 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:19:37.777 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:37.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.804+0000 7fa8eebac640 10 librados: watch_flush enter 2024-03-29T11:19:37.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.804+0000 7fa8eebac640 10 librados: watch_flush exit 2024-03-29T11:19:37.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.804+0000 7fa8eebac640 20 client.4231.objecter shutdown clearing up homeless session... 2024-03-29T11:19:37.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.804+0000 7fa8eebac640 10 client.4231.objecter successfully canceled tick 2024-03-29T11:19:37.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.804+0000 7fa8eebac640 1 -- 172.21.15.67:0/4064833693 >> v1:172.21.15.67:6816/404787197 conn(0x7fa8c806dc90 legacy=0x7fa8c8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:37.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.804+0000 7fa8eebac640 10 monclient: shutdown 2024-03-29T11:19:37.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.804+0000 7fa8eebac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:37.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.804+0000 7fa8eebac640 1 -- 172.21.15.67:0/4064833693 >> v1:172.21.15.82:6789/0 conn(0x7fa8e806d9d0 legacy=0x7fa8e80b2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:37.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.805+0000 7fa8eebac640 1 -- 172.21.15.67:0/4064833693 shutdown_connections 2024-03-29T11:19:37.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.805+0000 7fa8eebac640 1 -- 172.21.15.67:0/4064833693 >> 172.21.15.67:0/4064833693 conn(0x7fa8e806b660 msgr2=0x7fa8e80abdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:37.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.805+0000 7fa8eebac640 1 -- 172.21.15.67:0/4064833693 shutdown_connections 2024-03-29T11:19:37.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.805+0000 7fa8eebac640 1 -- 172.21.15.67:0/4064833693 wait complete. 2024-03-29T11:19:37.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:37.805+0000 7fa8eebac640 1 librados: shutdown 2024-03-29T11:19:37.797 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:44.604 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:44.765 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.788+0000 7efecc6d3640 10 monclient: get_monmap_and_config 2024-03-29T11:19:44.765 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.788+0000 7efecc6d3640 10 monclient: build_initial_monmap 2024-03-29T11:19:44.765 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.788+0000 7efecc6d3640 10 monclient: monmap: 2024-03-29T11:19:44.765 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:19:44.765 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:44.765 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:19:44.789103+0000 2024-03-29T11:19:44.765 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:19:44.789103+0000 2024-03-29T11:19:44.765 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:44.765 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:44.765 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:44.765 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:44.765 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:44.766 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:44.766 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.789+0000 7efecc6d3640 1 Processor -- start 2024-03-29T11:19:44.766 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.789+0000 7efecc6d3640 1 -- start start 2024-03-29T11:19:44.766 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.789+0000 7efecc6d3640 10 monclient: init 2024-03-29T11:19:44.767 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.790+0000 7efecc6d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:44.767 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.790+0000 7efecc6d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:44.767 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.790+0000 7efecc6d3640 10 monclient(hunting): picked mon.noname-c con 0x7efec41457f0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:44.767 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.790+0000 7efecc6d3640 10 monclient(hunting): picked mon.noname-a con 0x7efec413f650 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:44.767 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.790+0000 7efecc6d3640 10 monclient(hunting): picked mon.noname-b con 0x7efec40a9d70 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:44.767 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.790+0000 7efecc6d3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec40aaef0 con 0x7efec413f650 2024-03-29T11:19:44.767 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.790+0000 7efecc6d3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec4145bd0 con 0x7efec41457f0 2024-03-29T11:19:44.767 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.790+0000 7efecc6d3640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec4145eb0 con 0x7efec40a9d70 2024-03-29T11:19:44.768 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.790+0000 7efecc6d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:44.768 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.790+0000 7efecc6d3640 10 monclient(hunting): authenticate will time out at 845.086570s 2024-03-29T11:19:44.768 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.790+0000 7efeca46f640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7efec41457f0 0x7efec413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:50822/0 (socket says 172.21.15.67:50822) 2024-03-29T11:19:44.768 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.790+0000 7efeca46f640 1 -- 172.21.15.67:0/2946667602 learned_addr learned my addr 172.21.15.67:0/2946667602 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:19:44.769 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 1 -- 172.21.15.67:0/2946667602 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848445775 0 0) 0x7efec4145eb0 con 0x7efec40a9d70 2024-03-29T11:19:44.769 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient(hunting): _negotiate 2024-03-29T11:19:44.769 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:44.769 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:44.769 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:19:44.769 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 1 -- 172.21.15.67:0/2946667602 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb0003520 con 0x7efec40a9d70 2024-03-29T11:19:44.769 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 1 -- 172.21.15.67:0/2946667602 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211388107 0 0) 0x7efec40aaef0 con 0x7efec413f650 2024-03-29T11:19:44.769 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient(hunting): _negotiate 2024-03-29T11:19:44.769 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:44.769 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:44.769 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:19:44.769 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 1 -- 172.21.15.67:0/2946667602 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb0003840 con 0x7efec413f650 2024-03-29T11:19:44.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 1 -- 172.21.15.67:0/2946667602 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 147297314 0 0) 0x7efeb0003520 con 0x7efec40a9d70 2024-03-29T11:19:44.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 1 -- 172.21.15.67:0/2946667602 >> v1:172.21.15.82:6789/0 conn(0x7efec41457f0 legacy=0x7efec413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:44.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 1 -- 172.21.15.67:0/2946667602 >> v1:172.21.15.67:6789/0 conn(0x7efec413f650 legacy=0x7efec40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:44.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient: _finish_hunting 0 2024-03-29T11:19:44.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 1 monclient: found mon.noname-b 2024-03-29T11:19:44.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:44.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:19:44.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 1 -- 172.21.15.67:0/2946667602 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec4146340 con 0x7efec40a9d70 2024-03-29T11:19:44.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient: _finish_auth 0 2024-03-29T11:19:44.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient: _check_auth_tickets 2024-03-29T11:19:44.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:44.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efecc6d3640 5 monclient: authenticate success, global_id 4235 2024-03-29T11:19:44.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efecc6d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 1 -- 172.21.15.67:0/2946667602 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7efebc0019f0 con 0x7efec40a9d70 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efec946d640 10 monclient: dump: 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:44.771 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.792+0000 7efecc6d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:44.773 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.793+0000 7efec946d640 1 -- 172.21.15.67:0/2946667602 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efebc001bd0 con 0x7efec40a9d70 2024-03-29T11:19:44.773 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.793+0000 7efec946d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:44.773 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.793+0000 7efecc6d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:44.773 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.793+0000 7efecc6d3640 4 set_mon_vals no callback set 2024-03-29T11:19:44.773 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.793+0000 7efec946d640 1 -- 172.21.15.67:0/2946667602 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7efebc0019f0 con 0x7efec40a9d70 2024-03-29T11:19:44.773 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.793+0000 7efec946d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:44.773 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.793+0000 7efec946d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:44.773 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.793+0000 7efec946d640 10 monclient: dump: 2024-03-29T11:19:44.773 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:44.773 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:44.773 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:44.773 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:44.773 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:44.773 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:44.774 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:44.774 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:44.774 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:44.774 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:44.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.794+0000 7efecc6d3640 10 monclient: shutdown 2024-03-29T11:19:44.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.794+0000 7efecc6d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:44.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.794+0000 7efecc6d3640 1 -- 172.21.15.67:0/2946667602 >> v1:172.21.15.67:6790/0 conn(0x7efec40a9d70 legacy=0x7efec414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:44.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.794+0000 7efecc6d3640 1 -- 172.21.15.67:0/2946667602 shutdown_connections 2024-03-29T11:19:44.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.794+0000 7efecc6d3640 1 -- 172.21.15.67:0/2946667602 >> 172.21.15.67:0/2946667602 conn(0x7efec4066a00 msgr2=0x7efec4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:44.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.794+0000 7efecc6d3640 1 -- 172.21.15.67:0/2946667602 shutdown_connections 2024-03-29T11:19:44.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.795+0000 7efecc6d3640 1 -- 172.21.15.67:0/2946667602 wait complete. 2024-03-29T11:19:44.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.796+0000 7efecc6d3640 10 monclient: build_initial_monmap 2024-03-29T11:19:44.775 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.796+0000 7efecc6d3640 10 monclient: monmap: 2024-03-29T11:19:44.775 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:19:44.775 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:44.775 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:19:44.775 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:19:44.775 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:44.775 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:44.775 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:44.775 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:44.775 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:44.775 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:44.775 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.796+0000 7efecc6d3640 1 librados: starting msgr at 2024-03-29T11:19:44.775 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.796+0000 7efecc6d3640 1 librados: starting objecter 2024-03-29T11:19:44.775 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.796+0000 7efecc6d3640 1 Processor -- start 2024-03-29T11:19:44.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.796+0000 7efecc6d3640 1 -- start start 2024-03-29T11:19:44.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.796+0000 7efecc6d3640 1 librados: setting wanted keys 2024-03-29T11:19:44.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.796+0000 7efecc6d3640 1 librados: calling monclient init 2024-03-29T11:19:44.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.796+0000 7efecc6d3640 10 monclient: init 2024-03-29T11:19:44.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.797+0000 7efecc6d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:44.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.797+0000 7efecc6d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:44.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.797+0000 7efecc6d3640 10 monclient(hunting): picked mon.noname-a con 0x7efec40a9d70 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:44.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.797+0000 7efecc6d3640 10 monclient(hunting): picked mon.noname-c con 0x7efec413f650 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:44.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.797+0000 7efecc6d3640 10 monclient(hunting): picked mon.noname-b con 0x7efec41457f0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:44.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.797+0000 7efecc6d3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec40ac180 con 0x7efec40a9d70 2024-03-29T11:19:44.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.797+0000 7efecc6d3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec4144590 con 0x7efec41457f0 2024-03-29T11:19:44.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.797+0000 7efecc6d3640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec4144720 con 0x7efec413f650 2024-03-29T11:19:44.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.797+0000 7efecc6d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:44.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.797+0000 7efecc6d3640 10 monclient(hunting): authenticate will time out at 845.093539s 2024-03-29T11:19:44.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.797+0000 7efecac70640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7efec41457f0 0x7efec41dabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:51012/0 (socket says 172.21.15.67:51012) 2024-03-29T11:19:44.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.797+0000 7efecac70640 1 -- 172.21.15.67:0/643526328 learned_addr learned my addr 172.21.15.67:0/643526328 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:19:44.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 client.?.objecter ms_handle_connect 0x7efec40a9d70 2024-03-29T11:19:44.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:44.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 client.?.objecter ms_handle_connect 0x7efec413f650 2024-03-29T11:19:44.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:44.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138200562 0 0) 0x7efec40ac180 con 0x7efec40a9d70 2024-03-29T11:19:44.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:44.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:44.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:44.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:19:44.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea8004eb0 con 0x7efec40a9d70 2024-03-29T11:19:44.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 client.?.objecter ms_handle_connect 0x7efec41457f0 2024-03-29T11:19:44.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:44.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2708388912 0 0) 0x7efea8004eb0 con 0x7efec40a9d70 2024-03-29T11:19:44.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efec40ac180 con 0x7efec40a9d70 2024-03-29T11:19:44.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7efec00017d0 con 0x7efec40a9d70 2024-03-29T11:19:44.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:44.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:44.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 monclient(hunting): dump: 2024-03-29T11:19:44.781 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:44.782 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:44.782 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:44.782 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:44.782 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:44.782 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:44.782 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:44.782 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:44.782 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:44.782 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:44.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:44.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57813096 0 0) 0x7efec4144590 con 0x7efec41457f0 2024-03-29T11:19:44.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:44.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:44.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:44.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:19:44.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.798+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea8004eb0 con 0x7efec41457f0 2024-03-29T11:19:44.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1847983682 0 0) 0x7efec40ac180 con 0x7efec40a9d70 2024-03-29T11:19:44.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 >> v1:172.21.15.67:6790/0 conn(0x7efec413f650 legacy=0x7efec41da4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:44.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 >> v1:172.21.15.82:6789/0 conn(0x7efec41457f0 legacy=0x7efec41dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:44.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:44.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 1 monclient: found mon.a 2024-03-29T11:19:44.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:44.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:44.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec4141710 con 0x7efec40a9d70 2024-03-29T11:19:44.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:44.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:44.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:44.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efecc6d3640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:19:44.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efecc6d3640 10 monclient: _renew_subs 2024-03-29T11:19:44.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efecc6d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:44.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efecc6d3640 1 -- 172.21.15.67:0/643526328 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec41419c0 con 0x7efec40a9d70 2024-03-29T11:19:44.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efecc6d3640 10 client.4248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:44.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efecc6d3640 10 monclient: _renew_subs 2024-03-29T11:19:44.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efecc6d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:44.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efecc6d3640 1 -- 172.21.15.67:0/643526328 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efec4137850 con 0x7efec40a9d70 2024-03-29T11:19:44.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efecc6d3640 1 librados: init done 2024-03-29T11:19:44.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efec0001d20 con 0x7efec40a9d70 2024-03-29T11:19:44.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:44.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7efec0004980 con 0x7efec40a9d70 2024-03-29T11:19:44.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efec8c6c640 4 set_mon_vals no callback set 2024-03-29T11:19:44.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:44.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:44.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.799+0000 7efebaffd640 10 monclient: dump: 2024-03-29T11:19:44.785 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:44.785 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:44.785 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:44.785 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:44.785 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:44.785 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:44.785 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:44.785 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:44.785 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:44.785 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:44.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.801+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7efec001a950 con 0x7efec40a9d70 2024-03-29T11:19:44.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.802+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7efec0084990 con 0x7efec40a9d70 2024-03-29T11:19:44.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.802+0000 7efebaffd640 10 client.4248.objecter ms_dispatch 0x7efec4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:44.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.802+0000 7efebaffd640 3 client.4248.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:44.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.802+0000 7efebaffd640 3 client.4248.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:44.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.802+0000 7efebaffd640 20 client.4248.objecter dump_active .. 0 homeless 2024-03-29T11:19:44.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.802+0000 7efec9c6e640 10 monclient: get_auth_request con 0x7efea806ddc0 auth_method 0 2024-03-29T11:19:44.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.802+0000 7efecc6d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:44.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.802+0000 7efecc6d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:44.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.802+0000 7efebaffd640 10 client.4248.objecter ms_handle_connect 0x7efea806ddc0 2024-03-29T11:19:44.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.802+0000 7efecc6d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:44.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.803+0000 7efecc6d3640 1 -- 172.21.15.67:0/643526328 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeac005080 con 0x7efec40a9d70 2024-03-29T11:19:44.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.812+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efec00500a0 con 0x7efec40a9d70 2024-03-29T11:19:44.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.812+0000 7efebaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:44.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:44.812+0000 7efebaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:45.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.128+0000 7efecc6d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:45.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.128+0000 7efecc6d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:45.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.128+0000 7efecc6d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:19:45.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.128+0000 7efecc6d3640 1 -- 172.21.15.67:0/643526328 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efeac005330 con 0x7efec40a9d70 2024-03-29T11:19:45.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.130+0000 7efebaffd640 1 -- 172.21.15.67:0/643526328 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7efec00521a0 con 0x7efec40a9d70 2024-03-29T11:19:45.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.130+0000 7efebaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:45.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.130+0000 7efebaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:45.108 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:19:45.108 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:45.112 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.134+0000 7efecc6d3640 10 librados: watch_flush enter 2024-03-29T11:19:45.112 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.135+0000 7efecc6d3640 10 librados: watch_flush exit 2024-03-29T11:19:45.112 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.135+0000 7efecc6d3640 20 client.4248.objecter shutdown clearing up homeless session... 2024-03-29T11:19:45.112 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.135+0000 7efecc6d3640 10 client.4248.objecter successfully canceled tick 2024-03-29T11:19:45.112 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.135+0000 7efecc6d3640 1 -- 172.21.15.67:0/643526328 >> v1:172.21.15.67:6816/404787197 conn(0x7efea806ddc0 legacy=0x7efea8070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:45.112 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.135+0000 7efecc6d3640 10 monclient: shutdown 2024-03-29T11:19:45.112 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.135+0000 7efecc6d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:45.112 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.135+0000 7efecc6d3640 1 -- 172.21.15.67:0/643526328 >> v1:172.21.15.67:6789/0 conn(0x7efec40a9d70 legacy=0x7efec4136f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:45.112 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.135+0000 7efecc6d3640 1 -- 172.21.15.67:0/643526328 shutdown_connections 2024-03-29T11:19:45.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.135+0000 7efecc6d3640 1 -- 172.21.15.67:0/643526328 >> 172.21.15.67:0/643526328 conn(0x7efec4066a00 msgr2=0x7efec4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:45.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.135+0000 7efecc6d3640 1 -- 172.21.15.67:0/643526328 shutdown_connections 2024-03-29T11:19:45.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.135+0000 7efecc6d3640 1 -- 172.21.15.67:0/643526328 wait complete. 2024-03-29T11:19:45.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:45.136+0000 7efecc6d3640 1 librados: shutdown 2024-03-29T11:19:45.127 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:51.934 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:52.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.102+0000 7f344afc2640 10 monclient: get_monmap_and_config 2024-03-29T11:19:52.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.102+0000 7f344afc2640 10 monclient: build_initial_monmap 2024-03-29T11:19:52.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.102+0000 7f344afc2640 10 monclient: monmap: 2024-03-29T11:19:52.079 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:19:52.080 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:52.080 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:19:52.103519+0000 2024-03-29T11:19:52.080 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:19:52.103519+0000 2024-03-29T11:19:52.080 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:52.080 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:52.080 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:52.080 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:52.080 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:52.080 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:52.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.106+0000 7f344afc2640 1 Processor -- start 2024-03-29T11:19:52.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.107+0000 7f344afc2640 1 -- start start 2024-03-29T11:19:52.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.107+0000 7f344afc2640 10 monclient: init 2024-03-29T11:19:52.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.108+0000 7f344afc2640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:52.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.108+0000 7f344afc2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:52.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.108+0000 7f344afc2640 10 monclient(hunting): picked mon.noname-c con 0x7f34441422b0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:52.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.108+0000 7f344afc2640 10 monclient(hunting): picked mon.noname-a con 0x7f3444142690 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:52.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.108+0000 7f344afc2640 10 monclient(hunting): picked mon.noname-b con 0x7f34440b17d0 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:52.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.108+0000 7f344afc2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3444142c80 con 0x7f3444142690 2024-03-29T11:19:52.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.108+0000 7f344afc2640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34440aae40 con 0x7f34441422b0 2024-03-29T11:19:52.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.108+0000 7f344afc2640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f344413fbd0 con 0x7f34440b17d0 2024-03-29T11:19:52.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.108+0000 7f344afc2640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:52.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.108+0000 7f344afc2640 10 monclient(hunting): authenticate will time out at 852.404175s 2024-03-29T11:19:52.086 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.109+0000 7f3448d5e640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f34441422b0 0x7f344413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:54134/0 (socket says 172.21.15.67:54134) 2024-03-29T11:19:52.086 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.109+0000 7f3448d5e640 1 -- 172.21.15.67:0/3209053000 learned_addr learned my addr 172.21.15.67:0/3209053000 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:19:52.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.110+0000 7f34437fe640 1 -- 172.21.15.67:0/3209053000 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154289664 0 0) 0x7f34440aae40 con 0x7f34441422b0 2024-03-29T11:19:52.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.110+0000 7f34437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:52.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.110+0000 7f34437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:52.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.110+0000 7f34437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:52.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.110+0000 7f34437fe640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:19:52.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.110+0000 7f34437fe640 1 -- 172.21.15.67:0/3209053000 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f342c003520 con 0x7f34441422b0 2024-03-29T11:19:52.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.110+0000 7f34437fe640 1 -- 172.21.15.67:0/3209053000 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1516630839 0 0) 0x7f342c003520 con 0x7f34441422b0 2024-03-29T11:19:52.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.110+0000 7f34437fe640 1 -- 172.21.15.67:0/3209053000 >> v1:172.21.15.67:6790/0 conn(0x7f34440b17d0 legacy=0x7f34440b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:52.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.111+0000 7f34437fe640 1 -- 172.21.15.67:0/3209053000 >> v1:172.21.15.67:6789/0 conn(0x7f3444142690 legacy=0x7f34440b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:52.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f34437fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:52.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f34437fe640 1 monclient: found mon.noname-c 2024-03-29T11:19:52.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f34437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:52.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f34437fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:19:52.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f34437fe640 1 -- 172.21.15.67:0/3209053000 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3444151940 con 0x7f34441422b0 2024-03-29T11:19:52.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f34437fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:52.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f34437fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:52.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f34437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:52.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f34437fe640 1 -- 172.21.15.67:0/3209053000 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f3434001670 con 0x7f34441422b0 2024-03-29T11:19:52.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f344afc2640 5 monclient: authenticate success, global_id 4237 2024-03-29T11:19:52.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f344afc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:52.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f34437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:52.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f34437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:52.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f34437fe640 10 monclient: dump: 2024-03-29T11:19:52.093 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:52.093 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:52.093 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:52.093 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:52.093 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:52.093 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:52.093 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:52.093 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:52.093 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:52.093 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:52.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.114+0000 7f344afc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:52.094 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.116+0000 7f34437fe640 1 -- 172.21.15.67:0/3209053000 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34340018d0 con 0x7f34441422b0 2024-03-29T11:19:52.094 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.116+0000 7f34437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:52.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.116+0000 7f34437fe640 1 -- 172.21.15.67:0/3209053000 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f3434001ca0 con 0x7f34441422b0 2024-03-29T11:19:52.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.116+0000 7f344afc2640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:52.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.116+0000 7f344afc2640 4 set_mon_vals no callback set 2024-03-29T11:19:52.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.116+0000 7f34437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:52.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.116+0000 7f34437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:52.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.116+0000 7f34437fe640 10 monclient: dump: 2024-03-29T11:19:52.095 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:52.095 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:52.095 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:52.095 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:52.095 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:52.095 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:52.096 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:52.096 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:52.096 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:52.096 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:52.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.117+0000 7f344afc2640 10 monclient: shutdown 2024-03-29T11:19:52.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.117+0000 7f344afc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:52.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.117+0000 7f344afc2640 1 -- 172.21.15.67:0/3209053000 >> v1:172.21.15.82:6789/0 conn(0x7f34441422b0 legacy=0x7f344413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:52.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.117+0000 7f344afc2640 1 -- 172.21.15.67:0/3209053000 shutdown_connections 2024-03-29T11:19:52.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.117+0000 7f344afc2640 1 -- 172.21.15.67:0/3209053000 >> 172.21.15.67:0/3209053000 conn(0x7f3444066a00 msgr2=0x7f3444066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:52.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.117+0000 7f344afc2640 1 -- 172.21.15.67:0/3209053000 shutdown_connections 2024-03-29T11:19:52.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.117+0000 7f344afc2640 1 -- 172.21.15.67:0/3209053000 wait complete. 2024-03-29T11:19:52.098 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.119+0000 7f344afc2640 10 monclient: build_initial_monmap 2024-03-29T11:19:52.098 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.119+0000 7f344afc2640 10 monclient: monmap: 2024-03-29T11:19:52.098 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:19:52.098 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:52.098 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:19:52.099 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:19:52.099 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:52.099 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:52.099 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:52.099 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:52.099 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:52.099 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:52.099 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.120+0000 7f344afc2640 1 librados: starting msgr at 2024-03-29T11:19:52.099 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.120+0000 7f344afc2640 1 librados: starting objecter 2024-03-29T11:19:52.099 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.120+0000 7f344afc2640 1 Processor -- start 2024-03-29T11:19:52.099 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.120+0000 7f344afc2640 1 -- start start 2024-03-29T11:19:52.099 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.120+0000 7f344afc2640 1 librados: setting wanted keys 2024-03-29T11:19:52.099 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.120+0000 7f344afc2640 1 librados: calling monclient init 2024-03-29T11:19:52.100 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.120+0000 7f344afc2640 10 monclient: init 2024-03-29T11:19:52.100 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.121+0000 7f344afc2640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:52.100 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.121+0000 7f344afc2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:52.100 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.122+0000 7f344afc2640 10 monclient(hunting): picked mon.noname-b con 0x7f34440b17d0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:52.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.122+0000 7f344afc2640 10 monclient(hunting): picked mon.noname-a con 0x7f34441422b0 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:52.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.122+0000 7f344afc2640 10 monclient(hunting): picked mon.noname-c con 0x7f3444142690 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:52.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.122+0000 7f344afc2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34440ac5c0 con 0x7f34441422b0 2024-03-29T11:19:52.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.122+0000 7f344afc2640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34440ac8a0 con 0x7f34440b17d0 2024-03-29T11:19:52.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.122+0000 7f344afc2640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34440acb80 con 0x7f3444142690 2024-03-29T11:19:52.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.122+0000 7f344afc2640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:52.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.122+0000 7f344afc2640 10 monclient(hunting): authenticate will time out at 852.418697s 2024-03-29T11:19:52.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.122+0000 7f3448d5e640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f34440b17d0 0x7f344412fbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:54244/0 (socket says 172.21.15.67:54244) 2024-03-29T11:19:52.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.123+0000 7f3448d5e640 1 -- 172.21.15.67:0/4211420277 learned_addr learned my addr 172.21.15.67:0/4211420277 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:19:52.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.123+0000 7f34417fa640 10 client.?.objecter ms_handle_connect 0x7f34440b17d0 2024-03-29T11:19:52.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.123+0000 7f34417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:52.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.123+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496536644 0 0) 0x7f34440ac8a0 con 0x7f34440b17d0 2024-03-29T11:19:52.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.123+0000 7f34417fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:52.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.123+0000 7f34417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:52.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.123+0000 7f34417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:52.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.123+0000 7f34417fa640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:19:52.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.123+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3430004eb0 con 0x7f34440b17d0 2024-03-29T11:19:52.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.124+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2123455259 0 0) 0x7f3430004eb0 con 0x7f34440b17d0 2024-03-29T11:19:52.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.124+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3430005300 con 0x7f34440b17d0 2024-03-29T11:19:52.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.125+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f3434001be0 con 0x7f34440b17d0 2024-03-29T11:19:52.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.125+0000 7f34417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:52.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.125+0000 7f34417fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:52.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.125+0000 7f34417fa640 10 monclient(hunting): dump: 2024-03-29T11:19:52.108 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:52.108 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:52.108 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:52.108 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:52.108 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:52.108 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:52.108 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:52.108 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:52.108 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:52.108 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:52.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.125+0000 7f34417fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:52.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 11714894 0 0) 0x7f3430005300 con 0x7f34440b17d0 2024-03-29T11:19:52.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 >> v1:172.21.15.67:6790/0 conn(0x7f3444142690 legacy=0x7f34441e1a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:52.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 >> v1:172.21.15.67:6789/0 conn(0x7f34441422b0 legacy=0x7f3444130300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:52.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f34417fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:52.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f34417fa640 1 monclient: found mon.b 2024-03-29T11:19:52.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f34417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:52.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f34417fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:52.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34441320a0 con 0x7f34440b17d0 2024-03-29T11:19:52.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f34417fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:52.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f34417fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:52.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f34417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:52.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f344afc2640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:19:52.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f344afc2640 10 monclient: _renew_subs 2024-03-29T11:19:52.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f344afc2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:52.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.126+0000 7f344afc2640 1 -- 172.21.15.67:0/4211420277 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34440ac0e0 con 0x7f34440b17d0 2024-03-29T11:19:52.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.127+0000 7f344afc2640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:52.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.127+0000 7f344afc2640 10 monclient: _renew_subs 2024-03-29T11:19:52.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.127+0000 7f344afc2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:52.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.127+0000 7f344afc2640 1 -- 172.21.15.67:0/4211420277 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34440ac390 con 0x7f34440b17d0 2024-03-29T11:19:52.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.127+0000 7f344afc2640 1 librados: init done 2024-03-29T11:19:52.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.127+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3434004bf0 con 0x7f34440b17d0 2024-03-29T11:19:52.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.127+0000 7f34417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:52.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.127+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f3434001be0 con 0x7f34440b17d0 2024-03-29T11:19:52.111 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.127+0000 7f34417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:52.111 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.127+0000 7f34417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:52.111 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.127+0000 7f34417fa640 10 monclient: dump: 2024-03-29T11:19:52.111 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:52.111 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:52.111 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:52.111 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:52.111 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:52.111 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:52.111 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:52.111 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:52.111 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:52.111 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:52.112 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.127+0000 7f3442ffd640 4 set_mon_vals no callback set 2024-03-29T11:19:52.112 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.129+0000 7f344afc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:52.112 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.129+0000 7f344afc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:52.112 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.129+0000 7f344afc2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:52.112 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.129+0000 7f344afc2640 1 -- 172.21.15.67:0/4211420277 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3424005080 con 0x7f34440b17d0 2024-03-29T11:19:52.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.130+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f343401abd0 con 0x7f34440b17d0 2024-03-29T11:19:52.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.131+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f3434084a00 con 0x7f34440b17d0 2024-03-29T11:19:52.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.131+0000 7f34417fa640 10 client.4240.objecter ms_dispatch 0x7f3444000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:52.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.131+0000 7f34417fa640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:52.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.131+0000 7f34417fa640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:52.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.131+0000 7f34417fa640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-29T11:19:52.114 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.134+0000 7f3443fff640 10 monclient: get_auth_request con 0x7f343006dfa0 auth_method 0 2024-03-29T11:19:52.114 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.135+0000 7f34417fa640 10 client.4240.objecter ms_handle_connect 0x7f343006dfa0 2024-03-29T11:19:52.117 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.140+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3434050170 con 0x7f34440b17d0 2024-03-29T11:19:52.117 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.140+0000 7f34417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:52.117 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.140+0000 7f34417fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:52.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.472+0000 7f344afc2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:52.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.472+0000 7f344afc2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:52.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.472+0000 7f344afc2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:19:52.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.472+0000 7f344afc2640 1 -- 172.21.15.67:0/4211420277 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3424005330 con 0x7f34440b17d0 2024-03-29T11:19:52.452 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.475+0000 7f34417fa640 1 -- 172.21.15.67:0/4211420277 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f3434050170 con 0x7f34440b17d0 2024-03-29T11:19:52.452 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.475+0000 7f34417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:52.452 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.475+0000 7f34417fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:52.452 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:19:52.452 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:52.456 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.478+0000 7f344afc2640 10 librados: watch_flush enter 2024-03-29T11:19:52.456 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.479+0000 7f344afc2640 10 librados: watch_flush exit 2024-03-29T11:19:52.456 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.479+0000 7f344afc2640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-29T11:19:52.456 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.479+0000 7f344afc2640 10 client.4240.objecter successfully canceled tick 2024-03-29T11:19:52.456 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.479+0000 7f344afc2640 1 -- 172.21.15.67:0/4211420277 >> v1:172.21.15.67:6816/404787197 conn(0x7f343006dfa0 legacy=0x7f3430070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:52.457 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.479+0000 7f344afc2640 10 monclient: shutdown 2024-03-29T11:19:52.457 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.479+0000 7f344afc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:52.457 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.479+0000 7f344afc2640 1 -- 172.21.15.67:0/4211420277 >> v1:172.21.15.82:6789/0 conn(0x7f34440b17d0 legacy=0x7f344412fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:52.457 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.479+0000 7f344afc2640 1 -- 172.21.15.67:0/4211420277 shutdown_connections 2024-03-29T11:19:52.457 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.479+0000 7f344afc2640 1 -- 172.21.15.67:0/4211420277 >> 172.21.15.67:0/4211420277 conn(0x7f3444066a00 msgr2=0x7f344413c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:52.457 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.479+0000 7f344afc2640 1 -- 172.21.15.67:0/4211420277 shutdown_connections 2024-03-29T11:19:52.457 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.479+0000 7f344afc2640 1 -- 172.21.15.67:0/4211420277 wait complete. 2024-03-29T11:19:52.457 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:52.479+0000 7f344afc2640 1 librados: shutdown 2024-03-29T11:19:52.473 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:59.279 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:59.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.452+0000 7fa6b138c640 10 monclient: get_monmap_and_config 2024-03-29T11:19:59.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.452+0000 7fa6b138c640 10 monclient: build_initial_monmap 2024-03-29T11:19:59.430 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.452+0000 7fa6b138c640 10 monclient: monmap: 2024-03-29T11:19:59.430 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:19:59.430 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:59.430 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:19:59.454025+0000 2024-03-29T11:19:59.430 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:19:59.454025+0000 2024-03-29T11:19:59.430 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:59.430 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:59.430 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:59.430 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:19:59.430 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:19:59.430 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:59.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.453+0000 7fa6b138c640 1 Processor -- start 2024-03-29T11:19:59.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.453+0000 7fa6b138c640 1 -- start start 2024-03-29T11:19:59.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.453+0000 7fa6b138c640 10 monclient: init 2024-03-29T11:19:59.432 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.454+0000 7fa6b138c640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:59.437 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.454+0000 7fa6b138c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:59.437 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.454+0000 7fa6b138c640 10 monclient(hunting): picked mon.noname-b con 0x7fa6ac06d260 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:59.437 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.454+0000 7fa6b138c640 10 monclient(hunting): picked mon.noname-c con 0x7fa6ac0723b0 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:59.437 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.454+0000 7fa6b138c640 10 monclient(hunting): picked mon.noname-a con 0x7fa6ac146b80 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:59.437 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.454+0000 7fa6b138c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac0701d0 con 0x7fa6ac146b80 2024-03-29T11:19:59.437 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.454+0000 7fa6b138c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac070440 con 0x7fa6ac0723b0 2024-03-29T11:19:59.437 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.454+0000 7fa6b138c640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac06c2c0 con 0x7fa6ac06d260 2024-03-29T11:19:59.437 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.454+0000 7fa6b138c640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:59.437 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.454+0000 7fa6b138c640 10 monclient(hunting): authenticate will time out at 859.751068s 2024-03-29T11:19:59.437 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.457+0000 7fa6aaffd640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fa6ac06d260 0x7fa6ac071ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:59913/0 (socket says 172.21.15.67:59913) 2024-03-29T11:19:59.438 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.457+0000 7fa6aaffd640 1 -- 172.21.15.67:0/129822364 learned_addr learned my addr 172.21.15.67:0/129822364 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:19:59.438 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.457+0000 7fa6a9ffb640 1 -- 172.21.15.67:0/129822364 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809144682 0 0) 0x7fa6ac0701d0 con 0x7fa6ac146b80 2024-03-29T11:19:59.438 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.457+0000 7fa6a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.438 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.457+0000 7fa6a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.438 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.457+0000 7fa6a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.438 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.457+0000 7fa6a9ffb640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:19:59.438 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.457+0000 7fa6a9ffb640 1 -- 172.21.15.67:0/129822364 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa694003520 con 0x7fa6ac146b80 2024-03-29T11:19:59.438 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.457+0000 7fa6a9ffb640 1 -- 172.21.15.67:0/129822364 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018609983 0 0) 0x7fa6ac06c2c0 con 0x7fa6ac06d260 2024-03-29T11:19:59.438 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.457+0000 7fa6a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.438 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.457+0000 7fa6a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.438 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.457+0000 7fa6a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.438 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.457+0000 7fa6a9ffb640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:19:59.438 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.457+0000 7fa6a9ffb640 1 -- 172.21.15.67:0/129822364 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6ac0701d0 con 0x7fa6ac06d260 2024-03-29T11:19:59.439 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 1 -- 172.21.15.67:0/129822364 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 548799190 0 0) 0x7fa694003520 con 0x7fa6ac146b80 2024-03-29T11:19:59.439 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 1 -- 172.21.15.67:0/129822364 >> v1:172.21.15.67:6790/0 conn(0x7fa6ac06d260 legacy=0x7fa6ac071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.439 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 1 -- 172.21.15.67:0/129822364 >> v1:172.21.15.82:6789/0 conn(0x7fa6ac0723b0 legacy=0x7fa6ac146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:59.439 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:59.439 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 1 monclient: found mon.noname-a 2024-03-29T11:19:59.439 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:59.439 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:19:59.439 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 1 -- 172.21.15.67:0/129822364 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac06c7e0 con 0x7fa6ac146b80 2024-03-29T11:19:59.439 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:59.439 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:59.439 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:59.439 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6b138c640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:19:59.439 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6b138c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:59.439 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 1 -- 172.21.15.67:0/129822364 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa6a4001a10 con 0x7fa6ac146b80 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 10 monclient: dump: 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6b138c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 1 -- 172.21.15.67:0/129822364 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6a4001bf0 con 0x7fa6ac146b80 2024-03-29T11:19:59.440 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6b138c640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.458+0000 7fa6b138c640 4 set_mon_vals no callback set 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.459+0000 7fa6a9ffb640 1 -- 172.21.15.67:0/129822364 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa6a4001a10 con 0x7fa6ac146b80 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.459+0000 7fa6a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.459+0000 7fa6a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.459+0000 7fa6a9ffb640 10 monclient: dump: 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:59.441 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:59.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.459+0000 7fa6b138c640 10 monclient: shutdown 2024-03-29T11:19:59.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.459+0000 7fa6b138c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:59.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.459+0000 7fa6b138c640 1 -- 172.21.15.67:0/129822364 >> v1:172.21.15.67:6789/0 conn(0x7fa6ac146b80 legacy=0x7fa6ac151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.459+0000 7fa6b138c640 1 -- 172.21.15.67:0/129822364 shutdown_connections 2024-03-29T11:19:59.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.459+0000 7fa6b138c640 1 -- 172.21.15.67:0/129822364 >> 172.21.15.67:0/129822364 conn(0x7fa6ac067500 msgr2=0x7fa6ac0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:59.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.460+0000 7fa6b138c640 1 -- 172.21.15.67:0/129822364 shutdown_connections 2024-03-29T11:19:59.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.461+0000 7fa6b138c640 1 -- 172.21.15.67:0/129822364 wait complete. 2024-03-29T11:19:59.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.461+0000 7fa6b138c640 10 monclient: build_initial_monmap 2024-03-29T11:19:59.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.461+0000 7fa6b138c640 10 monclient: monmap: 2024-03-29T11:19:59.525 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:19:59.525 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:59.525 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:19:59.525 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:19:59.525 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:59.525 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:59.525 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:19:59.525 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:19:59.525 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:19:59.525 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:59.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.462+0000 7fa6b138c640 1 librados: starting msgr at 2024-03-29T11:19:59.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.462+0000 7fa6b138c640 1 librados: starting objecter 2024-03-29T11:19:59.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.462+0000 7fa6b138c640 1 Processor -- start 2024-03-29T11:19:59.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.462+0000 7fa6b138c640 1 -- start start 2024-03-29T11:19:59.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.462+0000 7fa6b138c640 1 librados: setting wanted keys 2024-03-29T11:19:59.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.462+0000 7fa6b138c640 1 librados: calling monclient init 2024-03-29T11:19:59.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.462+0000 7fa6b138c640 10 monclient: init 2024-03-29T11:19:59.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.463+0000 7fa6b138c640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:59.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.463+0000 7fa6b138c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:59.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.463+0000 7fa6b138c640 10 monclient(hunting): picked mon.noname-b con 0x7fa6ac06d260 addr v1:172.21.15.82:6789/0 2024-03-29T11:19:59.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.463+0000 7fa6b138c640 10 monclient(hunting): picked mon.noname-a con 0x7fa6ac0723b0 addr v1:172.21.15.67:6789/0 2024-03-29T11:19:59.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.465+0000 7fa6b138c640 10 monclient(hunting): picked mon.noname-c con 0x7fa6ac146b80 addr v1:172.21.15.67:6790/0 2024-03-29T11:19:59.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.465+0000 7fa6b138c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac0aac50 con 0x7fa6ac0723b0 2024-03-29T11:19:59.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.465+0000 7fa6b138c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac0aaf30 con 0x7fa6ac06d260 2024-03-29T11:19:59.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.465+0000 7fa6b138c640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac0ab210 con 0x7fa6ac146b80 2024-03-29T11:19:59.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.465+0000 7fa6b138c640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:59.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.465+0000 7fa6b138c640 10 monclient(hunting): authenticate will time out at 859.762188s 2024-03-29T11:19:59.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.466+0000 7fa6aaffd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fa6ac06d260 0x7fa6ac0ae990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:59282/0 (socket says 172.21.15.67:59282) 2024-03-29T11:19:59.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.466+0000 7fa6aaffd640 1 -- 172.21.15.67:0/811519016 learned_addr learned my addr 172.21.15.67:0/811519016 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:19:59.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.466+0000 7fa6937fe640 10 client.?.objecter ms_handle_connect 0x7fa6ac06d260 2024-03-29T11:19:59.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.466+0000 7fa6937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:59.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.467+0000 7fa6937fe640 10 client.?.objecter ms_handle_connect 0x7fa6ac146b80 2024-03-29T11:19:59.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.467+0000 7fa6937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:59.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.467+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554934272 0 0) 0x7fa6ac0ab210 con 0x7fa6ac146b80 2024-03-29T11:19:59.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.467+0000 7fa6937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:59.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.467+0000 7fa6937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:59.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.467+0000 7fa6937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:59.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.467+0000 7fa6937fe640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:19:59.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.467+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa698004eb0 con 0x7fa6ac146b80 2024-03-29T11:19:59.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2112968461 0 0) 0x7fa698004eb0 con 0x7fa6ac146b80 2024-03-29T11:19:59.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6ac0ab210 con 0x7fa6ac146b80 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa6a4001480 con 0x7fa6ac146b80 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 10 monclient(hunting): dump: 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:59.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 720992416 0 0) 0x7fa6ac0ab210 con 0x7fa6ac146b80 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 >> v1:172.21.15.82:6789/0 conn(0x7fa6ac06d260 legacy=0x7fa6ac0ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 >> v1:172.21.15.67:6789/0 conn(0x7fa6ac0723b0 legacy=0x7fa6ac1db890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 1 monclient: found mon.c 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac0a9e80 con 0x7fa6ac146b80 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.468+0000 7fa6937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6b138c640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6b138c640 10 monclient: _renew_subs 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6b138c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6b138c640 1 -- 172.21.15.67:0/811519016 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ac0a99c0 con 0x7fa6ac146b80 2024-03-29T11:19:59.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6b138c640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6b138c640 10 monclient: _renew_subs 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6b138c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6b138c640 1 -- 172.21.15.67:0/811519016 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ac14a5f0 con 0x7fa6ac146b80 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6b138c640 1 librados: init done 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6a4003c20 con 0x7fa6ac146b80 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa6a40020e0 con 0x7fa6ac146b80 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6937fe640 10 monclient: dump: 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:19:59.530 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.469+0000 7fa6a97fa640 4 set_mon_vals no callback set 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.470+0000 7fa6b138c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.470+0000 7fa6b138c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.470+0000 7fa6b138c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.470+0000 7fa6b138c640 1 -- 172.21.15.67:0/811519016 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa68c005080 con 0x7fa6ac146b80 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.471+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fa6a4003f30 con 0x7fa6ac146b80 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.471+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fa6a4083d10 con 0x7fa6ac146b80 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.471+0000 7fa6937fe640 10 client.4244.objecter ms_dispatch 0x7fa6ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.471+0000 7fa6937fe640 3 client.4244.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:59.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.471+0000 7fa6937fe640 3 client.4244.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:59.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.472+0000 7fa6937fe640 20 client.4244.objecter dump_active .. 0 homeless 2024-03-29T11:19:59.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.472+0000 7fa6aaffd640 10 monclient: get_auth_request con 0x7fa69806dcf0 auth_method 0 2024-03-29T11:19:59.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.472+0000 7fa6937fe640 10 client.4244.objecter ms_handle_connect 0x7fa69806dcf0 2024-03-29T11:19:59.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.482+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6a404f420 con 0x7fa6ac146b80 2024-03-29T11:19:59.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.482+0000 7fa6937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:59.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.482+0000 7fa6937fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:59.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.801+0000 7fa6b138c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:59.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.801+0000 7fa6b138c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:59.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.801+0000 7fa6b138c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:19:59.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.801+0000 7fa6b138c640 1 -- 172.21.15.67:0/811519016 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa68c005330 con 0x7fa6ac146b80 2024-03-29T11:19:59.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.802+0000 7fa6937fe640 1 -- 172.21.15.67:0/811519016 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fa6a4051520 con 0x7fa6ac146b80 2024-03-29T11:19:59.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.802+0000 7fa6937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:59.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.802+0000 7fa6937fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:59.780 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:19:59.781 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:59.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.806+0000 7fa6b138c640 10 librados: watch_flush enter 2024-03-29T11:19:59.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.806+0000 7fa6b138c640 10 librados: watch_flush exit 2024-03-29T11:19:59.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.806+0000 7fa6b138c640 20 client.4244.objecter shutdown clearing up homeless session... 2024-03-29T11:19:59.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.806+0000 7fa6b138c640 10 client.4244.objecter successfully canceled tick 2024-03-29T11:19:59.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.806+0000 7fa6b138c640 1 -- 172.21.15.67:0/811519016 >> v1:172.21.15.67:6816/404787197 conn(0x7fa69806dcf0 legacy=0x7fa6980701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.806+0000 7fa6b138c640 10 monclient: shutdown 2024-03-29T11:19:59.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.806+0000 7fa6b138c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:59.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.806+0000 7fa6b138c640 1 -- 172.21.15.67:0/811519016 >> v1:172.21.15.67:6790/0 conn(0x7fa6ac146b80 legacy=0x7fa6ac1dbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.807+0000 7fa6b138c640 1 -- 172.21.15.67:0/811519016 shutdown_connections 2024-03-29T11:19:59.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.807+0000 7fa6b138c640 1 -- 172.21.15.67:0/811519016 >> 172.21.15.67:0/811519016 conn(0x7fa6ac067500 msgr2=0x7fa6ac145d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:59.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.807+0000 7fa6b138c640 1 -- 172.21.15.67:0/811519016 shutdown_connections 2024-03-29T11:19:59.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.807+0000 7fa6b138c640 1 -- 172.21.15.67:0/811519016 wait complete. 2024-03-29T11:19:59.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:19:59.807+0000 7fa6b138c640 1 librados: shutdown 2024-03-29T11:19:59.802 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:06.609 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:06.767 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.790+0000 7f2623c74640 10 monclient: get_monmap_and_config 2024-03-29T11:20:06.767 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.790+0000 7f2623c74640 10 monclient: build_initial_monmap 2024-03-29T11:20:06.767 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.790+0000 7f2623c74640 10 monclient: monmap: 2024-03-29T11:20:06.767 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:06.767 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:06.767 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:20:06.791516+0000 2024-03-29T11:20:06.767 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:20:06.791516+0000 2024-03-29T11:20:06.767 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:06.768 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:06.768 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:06.768 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:20:06.768 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:20:06.768 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:06.769 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.791+0000 7f2623c74640 1 Processor -- start 2024-03-29T11:20:06.769 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.791+0000 7f2623c74640 1 -- start start 2024-03-29T11:20:06.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.791+0000 7f2623c74640 10 monclient: init 2024-03-29T11:20:06.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.792+0000 7f2623c74640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:06.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.792+0000 7f2623c74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:06.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.792+0000 7f2623c74640 10 monclient(hunting): picked mon.noname-c con 0x7f261c0708a0 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:06.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.792+0000 7f2623c74640 10 monclient(hunting): picked mon.noname-b con 0x7f261c06fe90 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:06.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.792+0000 7f2623c74640 10 monclient(hunting): picked mon.noname-a con 0x7f261c143c50 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:06.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.792+0000 7f2623c74640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261c072630 con 0x7f261c143c50 2024-03-29T11:20:06.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.792+0000 7f2623c74640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261c071410 con 0x7f261c0708a0 2024-03-29T11:20:06.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.792+0000 7f2623c74640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261c0afaa0 con 0x7f261c06fe90 2024-03-29T11:20:06.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.792+0000 7f2623c74640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:06.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.792+0000 7f2623c74640 10 monclient(hunting): authenticate will time out at 867.088701s 2024-03-29T11:20:06.771 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.792+0000 7f2621a10640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f261c0708a0 0x7f261c0af390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:40992/0 (socket says 172.21.15.67:40992) 2024-03-29T11:20:06.771 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.792+0000 7f2621a10640 1 -- 172.21.15.67:0/29908537 learned_addr learned my addr 172.21.15.67:0/29908537 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:06.771 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.794+0000 7f2620a0e640 1 -- 172.21.15.67:0/29908537 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019229783 0 0) 0x7f261c071410 con 0x7f261c0708a0 2024-03-29T11:20:06.771 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.794+0000 7f2620a0e640 10 monclient(hunting): _negotiate 2024-03-29T11:20:06.771 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.794+0000 7f2620a0e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:06.772 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.794+0000 7f2620a0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:06.772 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.794+0000 7f2620a0e640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:20:06.772 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.794+0000 7f2620a0e640 1 -- 172.21.15.67:0/29908537 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2604003520 con 0x7f261c0708a0 2024-03-29T11:20:06.772 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.794+0000 7f2620a0e640 1 -- 172.21.15.67:0/29908537 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3696477550 0 0) 0x7f2604003520 con 0x7f261c0708a0 2024-03-29T11:20:06.772 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.794+0000 7f2620a0e640 1 -- 172.21.15.67:0/29908537 >> v1:172.21.15.67:6790/0 conn(0x7f261c06fe90 legacy=0x7f261c143540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:06.775 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.796+0000 7f2620a0e640 1 -- 172.21.15.67:0/29908537 >> v1:172.21.15.67:6789/0 conn(0x7f261c143c50 legacy=0x7f261c14b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.775 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.796+0000 7f2620a0e640 10 monclient: _finish_hunting 0 2024-03-29T11:20:06.775 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.796+0000 7f2620a0e640 1 monclient: found mon.noname-c 2024-03-29T11:20:06.775 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.796+0000 7f2620a0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:06.775 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.796+0000 7f2620a0e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:20:06.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.796+0000 7f2620a0e640 1 -- 172.21.15.67:0/29908537 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c0ac8b0 con 0x7f261c0708a0 2024-03-29T11:20:06.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.796+0000 7f2620a0e640 10 monclient: _finish_auth 0 2024-03-29T11:20:06.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.796+0000 7f2620a0e640 10 monclient: _check_auth_tickets 2024-03-29T11:20:06.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.796+0000 7f2620a0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:06.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.796+0000 7f2620a0e640 1 -- 172.21.15.67:0/29908537 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f2610001670 con 0x7f261c0708a0 2024-03-29T11:20:06.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.796+0000 7f2620a0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:06.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.796+0000 7f2620a0e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:06.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.796+0000 7f2620a0e640 10 monclient: dump: 2024-03-29T11:20:06.776 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:06.776 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:06.776 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:06.776 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:06.777 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:06.777 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:06.777 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:06.777 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:06.777 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:06.777 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:06.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.797+0000 7f2623c74640 5 monclient: authenticate success, global_id 4243 2024-03-29T11:20:06.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.797+0000 7f2623c74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:06.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.797+0000 7f2620a0e640 1 -- 172.21.15.67:0/29908537 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26100018d0 con 0x7f261c0708a0 2024-03-29T11:20:06.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.797+0000 7f2620a0e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:06.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.797+0000 7f2623c74640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:06.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.797+0000 7f2623c74640 4 set_mon_vals no callback set 2024-03-29T11:20:06.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.797+0000 7f2620a0e640 1 -- 172.21.15.67:0/29908537 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f2610001d20 con 0x7f261c0708a0 2024-03-29T11:20:06.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.797+0000 7f2620a0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:06.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.797+0000 7f2620a0e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:06.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.797+0000 7f2620a0e640 10 monclient: dump: 2024-03-29T11:20:06.778 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:06.778 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:06.778 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:06.778 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:06.778 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:06.778 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:06.778 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:06.779 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:06.779 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:06.779 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:06.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.798+0000 7f2623c74640 10 monclient: shutdown 2024-03-29T11:20:06.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.798+0000 7f2623c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:06.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.798+0000 7f2623c74640 1 -- 172.21.15.67:0/29908537 >> v1:172.21.15.82:6789/0 conn(0x7f261c0708a0 legacy=0x7f261c0af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.798+0000 7f2623c74640 1 -- 172.21.15.67:0/29908537 shutdown_connections 2024-03-29T11:20:06.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.798+0000 7f2623c74640 1 -- 172.21.15.67:0/29908537 >> 172.21.15.67:0/29908537 conn(0x7f261c0675b0 msgr2=0x7f261c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:06.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.798+0000 7f2623c74640 1 -- 172.21.15.67:0/29908537 shutdown_connections 2024-03-29T11:20:06.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.799+0000 7f2623c74640 1 -- 172.21.15.67:0/29908537 wait complete. 2024-03-29T11:20:06.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.799+0000 7f2623c74640 10 monclient: build_initial_monmap 2024-03-29T11:20:06.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.799+0000 7f2623c74640 10 monclient: monmap: 2024-03-29T11:20:06.781 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:06.781 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:06.781 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:20:06.781 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:20:06.781 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:06.781 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:06.781 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:06.781 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:20:06.781 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:20:06.782 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:06.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.800+0000 7f2623c74640 1 librados: starting msgr at 2024-03-29T11:20:06.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.800+0000 7f2623c74640 1 librados: starting objecter 2024-03-29T11:20:06.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.800+0000 7f2623c74640 1 Processor -- start 2024-03-29T11:20:06.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.800+0000 7f2623c74640 1 -- start start 2024-03-29T11:20:06.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.800+0000 7f2623c74640 1 librados: setting wanted keys 2024-03-29T11:20:06.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.800+0000 7f2623c74640 1 librados: calling monclient init 2024-03-29T11:20:06.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.800+0000 7f2623c74640 10 monclient: init 2024-03-29T11:20:06.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.801+0000 7f2623c74640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:06.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.801+0000 7f2623c74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:06.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.801+0000 7f2623c74640 10 monclient(hunting): picked mon.noname-b con 0x7f261c06fe90 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:06.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.802+0000 7f2621a10640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f261c06fe90 0x7f261c0ab400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:41954/0 (socket says 172.21.15.67:41954) 2024-03-29T11:20:06.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.802+0000 7f2623c74640 10 monclient(hunting): picked mon.noname-c con 0x7f261c0708a0 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:06.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.802+0000 7f2623c74640 10 monclient(hunting): picked mon.noname-a con 0x7f261c143c50 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:06.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.802+0000 7f2623c74640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261c0abda0 con 0x7f261c143c50 2024-03-29T11:20:06.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.802+0000 7f2623c74640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261c1401d0 con 0x7f261c06fe90 2024-03-29T11:20:06.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.802+0000 7f2623c74640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261c140360 con 0x7f261c0708a0 2024-03-29T11:20:06.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.802+0000 7f2623c74640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:06.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.802+0000 7f2623c74640 10 monclient(hunting): authenticate will time out at 867.098528s 2024-03-29T11:20:06.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.802+0000 7f2621a10640 1 -- 172.21.15.67:0/1517316515 learned_addr learned my addr 172.21.15.67:0/1517316515 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:06.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.803+0000 7f261a7fc640 10 client.?.objecter ms_handle_connect 0x7f261c06fe90 2024-03-29T11:20:06.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.803+0000 7f261a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:06.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.804+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246703591 0 0) 0x7f261c1401d0 con 0x7f261c06fe90 2024-03-29T11:20:06.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.804+0000 7f261a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:06.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.804+0000 7f261a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:06.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.804+0000 7f261a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:06.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.804+0000 7f261a7fc640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:20:06.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.804+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2608004eb0 con 0x7f261c06fe90 2024-03-29T11:20:06.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.805+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2267342753 0 0) 0x7f2608004eb0 con 0x7f261c06fe90 2024-03-29T11:20:06.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.805+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f261c1401d0 con 0x7f261c06fe90 2024-03-29T11:20:06.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.805+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f2610001d20 con 0x7f261c06fe90 2024-03-29T11:20:06.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.805+0000 7f261a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:06.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.805+0000 7f261a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:06.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.805+0000 7f261a7fc640 10 monclient(hunting): dump: 2024-03-29T11:20:06.786 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:06.787 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:06.787 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:06.787 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:06.787 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:06.787 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:06.787 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:06.787 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:06.787 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:06.787 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:06.787 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.805+0000 7f261a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:06.787 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.805+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1959727422 0 0) 0x7f261c1401d0 con 0x7f261c06fe90 2024-03-29T11:20:06.787 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.805+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 >> v1:172.21.15.67:6790/0 conn(0x7f261c0708a0 legacy=0x7f261c13ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.805+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 >> v1:172.21.15.67:6789/0 conn(0x7f261c143c50 legacy=0x7f261c1d3110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:06.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f261a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:06.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f261a7fc640 1 monclient: found mon.b 2024-03-29T11:20:06.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f261a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:06.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f261a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:06.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c0aae00 con 0x7f261c06fe90 2024-03-29T11:20:06.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f261a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:06.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f261a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:06.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f261a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:06.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f2623c74640 5 monclient: authenticate success, global_id 4246 2024-03-29T11:20:06.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f2623c74640 10 monclient: _renew_subs 2024-03-29T11:20:06.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f2623c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:06.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f2623c74640 1 -- 172.21.15.67:0/1517316515 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f261c0ab050 con 0x7f261c06fe90 2024-03-29T11:20:06.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f2623c74640 10 client.4246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:06.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f2623c74640 10 monclient: _renew_subs 2024-03-29T11:20:06.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f2623c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:06.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f2623c74640 1 -- 172.21.15.67:0/1517316515 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f261c13d380 con 0x7f261c06fe90 2024-03-29T11:20:06.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.806+0000 7f2623c74640 1 librados: init done 2024-03-29T11:20:06.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.807+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2610004d20 con 0x7f261c06fe90 2024-03-29T11:20:06.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.807+0000 7f261a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:06.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.807+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f2610005030 con 0x7f261c06fe90 2024-03-29T11:20:06.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.807+0000 7f261a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:06.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.807+0000 7f261a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:06.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.807+0000 7f261a7fc640 10 monclient: dump: 2024-03-29T11:20:06.789 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:06.789 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:06.789 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:06.790 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:06.790 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:06.790 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:06.790 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:06.790 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:06.790 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:06.790 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:06.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.807+0000 7f261bfff640 4 set_mon_vals no callback set 2024-03-29T11:20:06.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.808+0000 7f2603fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:06.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.808+0000 7f2603fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:06.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.808+0000 7f2603fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:06.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.808+0000 7f2603fff640 1 -- 172.21.15.67:0/1517316515 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f8005080 con 0x7f261c06fe90 2024-03-29T11:20:06.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.809+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f261001b000 con 0x7f261c06fe90 2024-03-29T11:20:06.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.810+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f2610084e90 con 0x7f261c06fe90 2024-03-29T11:20:06.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.810+0000 7f261a7fc640 10 client.4246.objecter ms_dispatch 0x7f261c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:06.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.810+0000 7f261a7fc640 3 client.4246.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:06.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.810+0000 7f261a7fc640 3 client.4246.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:06.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.811+0000 7f261a7fc640 20 client.4246.objecter dump_active .. 0 homeless 2024-03-29T11:20:06.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.812+0000 7f262120f640 10 monclient: get_auth_request con 0x7f260806dbf0 auth_method 0 2024-03-29T11:20:06.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.814+0000 7f261a7fc640 10 client.4246.objecter ms_handle_connect 0x7f260806dbf0 2024-03-29T11:20:06.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.817+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2610050770 con 0x7f261c06fe90 2024-03-29T11:20:06.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.817+0000 7f261a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:06.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:06.817+0000 7f261a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:07.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.152+0000 7f2603fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:07.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.152+0000 7f2603fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:07.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.152+0000 7f2603fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:07.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.152+0000 7f2603fff640 1 -- 172.21.15.67:0/1517316515 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25f8005330 con 0x7f261c06fe90 2024-03-29T11:20:07.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.154+0000 7f261a7fc640 1 -- 172.21.15.67:0/1517316515 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f26100526a0 con 0x7f261c06fe90 2024-03-29T11:20:07.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.154+0000 7f261a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:07.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.154+0000 7f261a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:07.131 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:20:07.131 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:07.135 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.158+0000 7f2603fff640 10 librados: watch_flush enter 2024-03-29T11:20:07.135 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.158+0000 7f2603fff640 10 librados: watch_flush exit 2024-03-29T11:20:07.135 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.158+0000 7f2603fff640 20 client.4246.objecter shutdown clearing up homeless session... 2024-03-29T11:20:07.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.158+0000 7f2603fff640 10 client.4246.objecter successfully canceled tick 2024-03-29T11:20:07.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.158+0000 7f2603fff640 1 -- 172.21.15.67:0/1517316515 >> v1:172.21.15.67:6816/404787197 conn(0x7f260806dbf0 legacy=0x7f26080700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:07.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.158+0000 7f2603fff640 10 monclient: shutdown 2024-03-29T11:20:07.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.158+0000 7f2603fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:07.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.158+0000 7f2603fff640 1 -- 172.21.15.67:0/1517316515 >> v1:172.21.15.82:6789/0 conn(0x7f261c06fe90 legacy=0x7f261c0ab400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:07.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.158+0000 7f2603fff640 1 -- 172.21.15.67:0/1517316515 shutdown_connections 2024-03-29T11:20:07.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.158+0000 7f2603fff640 1 -- 172.21.15.67:0/1517316515 >> 172.21.15.67:0/1517316515 conn(0x7f261c0675b0 msgr2=0x7f261c0b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:07.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.159+0000 7f2603fff640 1 -- 172.21.15.67:0/1517316515 shutdown_connections 2024-03-29T11:20:07.137 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.159+0000 7f2603fff640 1 -- 172.21.15.67:0/1517316515 wait complete. 2024-03-29T11:20:07.137 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:07.159+0000 7f2603fff640 1 librados: shutdown 2024-03-29T11:20:07.152 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:13.957 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:14.118 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.142+0000 7fa9a11e8640 10 monclient: get_monmap_and_config 2024-03-29T11:20:14.119 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.142+0000 7fa9a11e8640 10 monclient: build_initial_monmap 2024-03-29T11:20:14.119 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.142+0000 7fa9a11e8640 10 monclient: monmap: 2024-03-29T11:20:14.119 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:14.119 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:14.119 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:20:14.142823+0000 2024-03-29T11:20:14.119 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:20:14.142823+0000 2024-03-29T11:20:14.119 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:14.119 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:14.119 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:14.119 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:20:14.119 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:20:14.119 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:14.121 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.144+0000 7fa9a11e8640 1 Processor -- start 2024-03-29T11:20:14.125 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.145+0000 7fa9a11e8640 1 -- start start 2024-03-29T11:20:14.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.145+0000 7fa9a11e8640 10 monclient: init 2024-03-29T11:20:14.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.146+0000 7fa9a11e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:14.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.146+0000 7fa9a11e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:14.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.146+0000 7fa9a11e8640 10 monclient(hunting): picked mon.noname-a con 0x7fa99c0b1f60 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:14.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.146+0000 7fa9a11e8640 10 monclient(hunting): picked mon.noname-c con 0x7fa99c071eb0 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:14.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.146+0000 7fa9a11e8640 10 monclient(hunting): picked mon.noname-b con 0x7fa99c143d10 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:14.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.146+0000 7fa9a11e8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c06ef80 con 0x7fa99c0b1f60 2024-03-29T11:20:14.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.146+0000 7fa9a11e8640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c06f260 con 0x7fa99c071eb0 2024-03-29T11:20:14.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.146+0000 7fa9a11e8640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c0b40c0 con 0x7fa99c143d10 2024-03-29T11:20:14.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.146+0000 7fa9a11e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:14.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.146+0000 7fa9a11e8640 10 monclient(hunting): authenticate will time out at 874.442109s 2024-03-29T11:20:14.127 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.151+0000 7fa99ad9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa99c0b1f60 0x7fa99c143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45933/0 (socket says 172.21.15.67:45933) 2024-03-29T11:20:14.127 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.151+0000 7fa99ad9d640 1 -- 172.21.15.67:0/3154147819 learned_addr learned my addr 172.21.15.67:0/3154147819 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:14.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.151+0000 7fa999d9b640 1 -- 172.21.15.67:0/3154147819 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061809095 0 0) 0x7fa99c0b40c0 con 0x7fa99c143d10 2024-03-29T11:20:14.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.151+0000 7fa999d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:14.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.151+0000 7fa999d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:14.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.151+0000 7fa999d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:14.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.151+0000 7fa999d9b640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:20:14.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.151+0000 7fa999d9b640 1 -- 172.21.15.67:0/3154147819 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa984003520 con 0x7fa99c143d10 2024-03-29T11:20:14.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.151+0000 7fa999d9b640 1 -- 172.21.15.67:0/3154147819 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146855532 0 0) 0x7fa99c06ef80 con 0x7fa99c0b1f60 2024-03-29T11:20:14.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.151+0000 7fa999d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:14.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.151+0000 7fa999d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:14.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.151+0000 7fa999d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:14.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.151+0000 7fa999d9b640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:20:14.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.151+0000 7fa999d9b640 1 -- 172.21.15.67:0/3154147819 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa99c0b40c0 con 0x7fa99c0b1f60 2024-03-29T11:20:14.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 1 -- 172.21.15.67:0/3154147819 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 153562997 0 0) 0x7fa99c0b40c0 con 0x7fa99c0b1f60 2024-03-29T11:20:14.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 1 -- 172.21.15.67:0/3154147819 >> v1:172.21.15.67:6790/0 conn(0x7fa99c143d10 legacy=0x7fa99c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:14.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 1 -- 172.21.15.67:0/3154147819 >> v1:172.21.15.82:6789/0 conn(0x7fa99c071eb0 legacy=0x7fa99c146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:14.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:20:14.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 1 monclient: found mon.noname-a 2024-03-29T11:20:14.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:14.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:20:14.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 1 -- 172.21.15.67:0/3154147819 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c155d70 con 0x7fa99c0b1f60 2024-03-29T11:20:14.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 10 monclient: _finish_auth 0 2024-03-29T11:20:14.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:20:14.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:14.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa9a11e8640 5 monclient: authenticate success, global_id 4257 2024-03-29T11:20:14.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa9a11e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:14.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 1 -- 172.21.15.67:0/3154147819 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa9900015a0 con 0x7fa99c0b1f60 2024-03-29T11:20:14.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:14.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:14.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.152+0000 7fa999d9b640 10 monclient: dump: 2024-03-29T11:20:14.132 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:14.132 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:14.132 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:14.132 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:14.133 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:14.133 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:14.133 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:14.133 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:14.133 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:14.133 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:14.133 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.153+0000 7fa9a11e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:14.135 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.154+0000 7fa999d9b640 1 -- 172.21.15.67:0/3154147819 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9900017c0 con 0x7fa99c0b1f60 2024-03-29T11:20:14.135 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.154+0000 7fa999d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:14.135 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.154+0000 7fa999d9b640 1 -- 172.21.15.67:0/3154147819 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa990001b50 con 0x7fa99c0b1f60 2024-03-29T11:20:14.135 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.154+0000 7fa999d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:14.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.154+0000 7fa999d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:14.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.154+0000 7fa999d9b640 10 monclient: dump: 2024-03-29T11:20:14.136 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:14.136 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:14.136 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:14.136 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:14.136 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:14.136 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:14.136 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:14.136 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:14.136 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:14.136 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:14.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.154+0000 7fa9a11e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:14.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.154+0000 7fa9a11e8640 4 set_mon_vals no callback set 2024-03-29T11:20:14.137 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.155+0000 7fa9a11e8640 10 monclient: shutdown 2024-03-29T11:20:14.137 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.155+0000 7fa9a11e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:14.137 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.155+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3154147819 >> v1:172.21.15.67:6789/0 conn(0x7fa99c0b1f60 legacy=0x7fa99c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:14.137 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.155+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3154147819 shutdown_connections 2024-03-29T11:20:14.137 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.155+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3154147819 >> 172.21.15.67:0/3154147819 conn(0x7fa99c066a00 msgr2=0x7fa99c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:14.137 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.156+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3154147819 shutdown_connections 2024-03-29T11:20:14.137 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.156+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3154147819 wait complete. 2024-03-29T11:20:14.137 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.158+0000 7fa9a11e8640 10 monclient: build_initial_monmap 2024-03-29T11:20:14.137 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.158+0000 7fa9a11e8640 10 monclient: monmap: 2024-03-29T11:20:14.137 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:14.137 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:14.137 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:20:14.137 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:20:14.137 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:14.138 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:14.138 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:14.138 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:20:14.138 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:20:14.138 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:14.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.158+0000 7fa9a11e8640 1 librados: starting msgr at 2024-03-29T11:20:14.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.158+0000 7fa9a11e8640 1 librados: starting objecter 2024-03-29T11:20:14.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.158+0000 7fa9a11e8640 1 Processor -- start 2024-03-29T11:20:14.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.158+0000 7fa9a11e8640 1 -- start start 2024-03-29T11:20:14.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.158+0000 7fa9a11e8640 1 librados: setting wanted keys 2024-03-29T11:20:14.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.159+0000 7fa9a11e8640 1 librados: calling monclient init 2024-03-29T11:20:14.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.159+0000 7fa9a11e8640 10 monclient: init 2024-03-29T11:20:14.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.159+0000 7fa9a11e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:14.140 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.160+0000 7fa9a11e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:14.140 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.160+0000 7fa9a11e8640 10 monclient(hunting): picked mon.noname-a con 0x7fa99c143d10 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:14.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.164+0000 7fa9a11e8640 10 monclient(hunting): picked mon.noname-c con 0x7fa99c0700e0 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:14.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.164+0000 7fa99a59c640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fa99c0700e0 0x7fa99c1df0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:39713/0 (socket says 172.21.15.67:39713) 2024-03-29T11:20:14.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.164+0000 7fa99a59c640 1 -- 172.21.15.67:0/3253535880 learned_addr learned my addr 172.21.15.67:0/3253535880 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:14.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.164+0000 7fa9a11e8640 10 monclient(hunting): picked mon.noname-b con 0x7fa99c154a00 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:14.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.164+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3253535880 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c0715c0 con 0x7fa99c143d10 2024-03-29T11:20:14.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.164+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3253535880 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c0705a0 con 0x7fa99c154a00 2024-03-29T11:20:14.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.164+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3253535880 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c06fa50 con 0x7fa99c0700e0 2024-03-29T11:20:14.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.164+0000 7fa9a11e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:14.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.164+0000 7fa9a11e8640 10 monclient(hunting): authenticate will time out at 874.460656s 2024-03-29T11:20:14.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.165+0000 7fa9837fe640 10 client.?.objecter ms_handle_connect 0x7fa99c0700e0 2024-03-29T11:20:14.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.165+0000 7fa9837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:14.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.165+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935351421 0 0) 0x7fa99c06fa50 con 0x7fa99c0700e0 2024-03-29T11:20:14.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.165+0000 7fa9837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:14.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.165+0000 7fa9837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:14.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.165+0000 7fa9837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:14.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.165+0000 7fa9837fe640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:20:14.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.165+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa988004eb0 con 0x7fa99c0700e0 2024-03-29T11:20:14.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.170+0000 7fa9837fe640 10 client.?.objecter ms_handle_connect 0x7fa99c143d10 2024-03-29T11:20:14.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.170+0000 7fa9837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:14.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.170+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 140737725 0 0) 0x7fa988004eb0 con 0x7fa99c0700e0 2024-03-29T11:20:14.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.170+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa99c06fa50 con 0x7fa99c0700e0 2024-03-29T11:20:14.148 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.171+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548586034 0 0) 0x7fa99c0715c0 con 0x7fa99c143d10 2024-03-29T11:20:14.148 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.171+0000 7fa9837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:14.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:14.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:14.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:20:14.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa988004eb0 con 0x7fa99c143d10 2024-03-29T11:20:14.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 10 client.?.objecter ms_handle_connect 0x7fa99c154a00 2024-03-29T11:20:14.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:14.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa994001bf0 con 0x7fa99c0700e0 2024-03-29T11:20:14.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:14.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:14.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 10 monclient(hunting): dump: 2024-03-29T11:20:14.154 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:14.154 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:14.155 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:14.155 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:14.155 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:14.155 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:14.155 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:14.155 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:14.155 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:14.155 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:14.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:14.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2594424386 0 0) 0x7fa99c06fa50 con 0x7fa99c0700e0 2024-03-29T11:20:14.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 >> v1:172.21.15.82:6789/0 conn(0x7fa99c154a00 legacy=0x7fa99c154eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:14.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 >> v1:172.21.15.67:6789/0 conn(0x7fa99c143d10 legacy=0x7fa99c1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:14.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 1 monclient: found mon.c 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c06fd30 con 0x7fa99c0700e0 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9a11e8640 5 monclient: authenticate success, global_id 4250 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9a11e8640 10 monclient: _renew_subs 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9a11e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3253535880 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c1df7c0 con 0x7fa99c0700e0 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9a11e8640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9a11e8640 10 monclient: _renew_subs 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9a11e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:14.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3253535880 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c1557a0 con 0x7fa99c0700e0 2024-03-29T11:20:14.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.173+0000 7fa9a11e8640 1 librados: init done 2024-03-29T11:20:14.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.174+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa994001d50 con 0x7fa99c0700e0 2024-03-29T11:20:14.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.174+0000 7fa9837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:14.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.174+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa994004e20 con 0x7fa99c0700e0 2024-03-29T11:20:14.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.174+0000 7fa9837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:14.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.174+0000 7fa9837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:14.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.174+0000 7fa9837fe640 10 monclient: dump: 2024-03-29T11:20:14.157 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:14.157 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:14.157 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:14.157 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:14.157 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:14.157 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:14.157 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:14.158 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:14.158 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:14.158 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:14.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.174+0000 7fa99959a640 4 set_mon_vals no callback set 2024-03-29T11:20:14.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.175+0000 7fa9a11e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:14.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.175+0000 7fa9a11e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:14.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.175+0000 7fa9a11e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:14.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.175+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3253535880 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa97c005080 con 0x7fa99c0700e0 2024-03-29T11:20:14.159 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.179+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fa99401adf0 con 0x7fa99c0700e0 2024-03-29T11:20:14.159 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.180+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fa994084d80 con 0x7fa99c0700e0 2024-03-29T11:20:14.159 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.180+0000 7fa9837fe640 10 client.4250.objecter ms_dispatch 0x7fa99c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:14.159 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.180+0000 7fa9837fe640 3 client.4250.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:14.159 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.180+0000 7fa9837fe640 3 client.4250.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:14.160 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.180+0000 7fa9837fe640 20 client.4250.objecter dump_active .. 0 homeless 2024-03-29T11:20:14.160 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.181+0000 7fa99ad9d640 10 monclient: get_auth_request con 0x7fa98806dcb0 auth_method 0 2024-03-29T11:20:14.160 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.181+0000 7fa9837fe640 10 client.4250.objecter ms_handle_connect 0x7fa98806dcb0 2024-03-29T11:20:14.164 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.188+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa994050490 con 0x7fa99c0700e0 2024-03-29T11:20:14.164 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.188+0000 7fa9837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:14.165 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.188+0000 7fa9837fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:14.486 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.509+0000 7fa9a11e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:14.487 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.509+0000 7fa9a11e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:14.487 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.509+0000 7fa9a11e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:14.487 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.509+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3253535880 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa97c005330 con 0x7fa99c0700e0 2024-03-29T11:20:14.487 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.510+0000 7fa9837fe640 1 -- 172.21.15.67:0/3253535880 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fa994052590 con 0x7fa99c0700e0 2024-03-29T11:20:14.487 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.510+0000 7fa9837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:14.487 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.510+0000 7fa9837fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:14.487 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:20:14.488 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:14.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.515+0000 7fa9a11e8640 10 librados: watch_flush enter 2024-03-29T11:20:14.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.516+0000 7fa9a11e8640 10 librados: watch_flush exit 2024-03-29T11:20:14.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.516+0000 7fa9a11e8640 20 client.4250.objecter shutdown clearing up homeless session... 2024-03-29T11:20:14.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.516+0000 7fa9a11e8640 10 client.4250.objecter successfully canceled tick 2024-03-29T11:20:14.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.516+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3253535880 >> v1:172.21.15.67:6816/404787197 conn(0x7fa98806dcb0 legacy=0x7fa988070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:14.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.516+0000 7fa9a11e8640 10 monclient: shutdown 2024-03-29T11:20:14.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.516+0000 7fa9a11e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:14.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.516+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3253535880 >> v1:172.21.15.67:6790/0 conn(0x7fa99c0700e0 legacy=0x7fa99c1df0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:14.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.516+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3253535880 shutdown_connections 2024-03-29T11:20:14.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.516+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3253535880 >> 172.21.15.67:0/3253535880 conn(0x7fa99c066a00 msgr2=0x7fa99c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:14.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.517+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3253535880 shutdown_connections 2024-03-29T11:20:14.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.517+0000 7fa9a11e8640 1 -- 172.21.15.67:0/3253535880 wait complete. 2024-03-29T11:20:14.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:14.517+0000 7fa9a11e8640 1 librados: shutdown 2024-03-29T11:20:14.512 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:21.320 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:21.472 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.495+0000 7f4cf7805640 10 monclient: get_monmap_and_config 2024-03-29T11:20:21.472 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.495+0000 7f4cf7805640 10 monclient: build_initial_monmap 2024-03-29T11:20:21.472 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.495+0000 7f4cf7805640 10 monclient: monmap: 2024-03-29T11:20:21.472 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:21.472 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:21.472 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:20:21.496022+0000 2024-03-29T11:20:21.472 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:20:21.496022+0000 2024-03-29T11:20:21.472 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:21.472 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:21.472 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:21.472 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:20:21.473 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:20:21.473 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:21.474 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.497+0000 7f4cf7805640 1 Processor -- start 2024-03-29T11:20:21.474 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.497+0000 7f4cf7805640 1 -- start start 2024-03-29T11:20:21.474 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.497+0000 7f4cf7805640 10 monclient: init 2024-03-29T11:20:21.476 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.498+0000 7f4cf7805640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:21.476 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.498+0000 7f4cf7805640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:21.476 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.498+0000 7f4cf7805640 10 monclient(hunting): picked mon.noname-c con 0x7f4ce8037220 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:21.476 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.498+0000 7f4cf7805640 10 monclient(hunting): picked mon.noname-b con 0x7f4ce80cf8a0 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:21.476 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.498+0000 7f4cf7805640 10 monclient(hunting): picked mon.noname-a con 0x7f4ce802b0f0 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:21.476 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.498+0000 7f4cf7805640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce802c120 con 0x7f4ce802b0f0 2024-03-29T11:20:21.476 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.498+0000 7f4cf7805640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce802c400 con 0x7f4ce8037220 2024-03-29T11:20:21.476 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.498+0000 7f4cf7805640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce802abd0 con 0x7f4ce80cf8a0 2024-03-29T11:20:21.476 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.498+0000 7f4cf7805640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:21.477 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.498+0000 7f4cf7805640 10 monclient(hunting): authenticate will time out at 881.794322s 2024-03-29T11:20:21.479 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.502+0000 7f4cf5da2640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4ce802b0f0 0x7f4ce80d7530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34629/0 (socket says 172.21.15.67:34629) 2024-03-29T11:20:21.479 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.502+0000 7f4cf5da2640 1 -- 172.21.15.67:0/461712738 learned_addr learned my addr 172.21.15.67:0/461712738 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:21.479 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.502+0000 7f4ce7fff640 1 -- 172.21.15.67:0/461712738 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510603834 0 0) 0x7f4ce802c120 con 0x7f4ce802b0f0 2024-03-29T11:20:21.479 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.502+0000 7f4ce7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:21.479 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.502+0000 7f4ce7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:21.479 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.502+0000 7f4ce7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:21.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.502+0000 7f4ce7fff640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:20:21.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.502+0000 7f4ce7fff640 1 -- 172.21.15.67:0/461712738 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cdc003520 con 0x7f4ce802b0f0 2024-03-29T11:20:21.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.502+0000 7f4ce7fff640 1 -- 172.21.15.67:0/461712738 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1270937759 0 0) 0x7f4cdc003520 con 0x7f4ce802b0f0 2024-03-29T11:20:21.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.502+0000 7f4ce7fff640 1 -- 172.21.15.67:0/461712738 >> v1:172.21.15.67:6790/0 conn(0x7f4ce80cf8a0 legacy=0x7f4ce80c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:21.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.502+0000 7f4ce7fff640 1 -- 172.21.15.67:0/461712738 >> v1:172.21.15.82:6789/0 conn(0x7f4ce8037220 legacy=0x7f4ce80cf190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:21.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:21.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 1 monclient: found mon.noname-a 2024-03-29T11:20:21.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:21.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:20:21.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 1 -- 172.21.15.67:0/461712738 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce80c92c0 con 0x7f4ce802b0f0 2024-03-29T11:20:21.481 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 10 monclient: _finish_auth 0 2024-03-29T11:20:21.481 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:21.481 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:21.481 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 1 -- 172.21.15.67:0/461712738 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f4cf004f730 con 0x7f4ce802b0f0 2024-03-29T11:20:21.481 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:21.481 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:21.481 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 10 monclient: dump: 2024-03-29T11:20:21.481 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:21.481 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:21.481 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:21.481 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:21.481 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:21.481 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:21.482 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:21.482 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:21.482 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:21.482 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:21.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4cf7805640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:20:21.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4cf7805640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:21.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 1 -- 172.21.15.67:0/461712738 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cf0075f10 con 0x7f4ce802b0f0 2024-03-29T11:20:21.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:21.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4cf7805640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:21.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4cf7805640 4 set_mon_vals no callback set 2024-03-29T11:20:21.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 1 -- 172.21.15.67:0/461712738 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f4cf00758b0 con 0x7f4ce802b0f0 2024-03-29T11:20:21.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:21.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:21.484 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.503+0000 7f4ce7fff640 10 monclient: dump: 2024-03-29T11:20:21.484 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:21.484 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:21.484 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:21.484 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:21.484 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:21.484 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:21.484 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:21.484 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:21.484 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:21.484 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:21.484 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.504+0000 7f4cf7805640 10 monclient: shutdown 2024-03-29T11:20:21.485 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.504+0000 7f4cf7805640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:21.485 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.504+0000 7f4cf7805640 1 -- 172.21.15.67:0/461712738 >> v1:172.21.15.67:6789/0 conn(0x7f4ce802b0f0 legacy=0x7f4ce80d7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:21.485 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.504+0000 7f4cf7805640 1 -- 172.21.15.67:0/461712738 shutdown_connections 2024-03-29T11:20:21.485 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.504+0000 7f4cf7805640 1 -- 172.21.15.67:0/461712738 >> 172.21.15.67:0/461712738 conn(0x7f4ce803a4b0 msgr2=0x7f4ce80367f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:21.485 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.506+0000 7f4cf7805640 1 -- 172.21.15.67:0/461712738 shutdown_connections 2024-03-29T11:20:21.486 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.506+0000 7f4cf7805640 1 -- 172.21.15.67:0/461712738 wait complete. 2024-03-29T11:20:21.486 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.507+0000 7f4cf7805640 10 monclient: build_initial_monmap 2024-03-29T11:20:21.486 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.507+0000 7f4cf7805640 10 monclient: monmap: 2024-03-29T11:20:21.486 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:21.486 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:21.486 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:20:21.487 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:20:21.487 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:21.487 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:21.487 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:21.487 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:20:21.487 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:20:21.487 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:21.487 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.508+0000 7f4cf7805640 1 librados: starting msgr at 2024-03-29T11:20:21.487 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.508+0000 7f4cf7805640 1 librados: starting objecter 2024-03-29T11:20:21.487 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.508+0000 7f4cf7805640 1 Processor -- start 2024-03-29T11:20:21.487 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.508+0000 7f4cf7805640 1 -- start start 2024-03-29T11:20:21.487 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.508+0000 7f4cf7805640 1 librados: setting wanted keys 2024-03-29T11:20:21.488 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.508+0000 7f4cf7805640 1 librados: calling monclient init 2024-03-29T11:20:21.488 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.508+0000 7f4cf7805640 10 monclient: init 2024-03-29T11:20:21.488 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.509+0000 7f4cf7805640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:21.488 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.509+0000 7f4cf7805640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:21.488 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.509+0000 7f4cf7805640 10 monclient(hunting): picked mon.noname-b con 0x7f4ce802b0f0 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:21.488 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.512+0000 7f4cf7805640 10 monclient(hunting): picked mon.noname-c con 0x7f4ce8037220 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:21.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.512+0000 7f4cf7805640 10 monclient(hunting): picked mon.noname-a con 0x7f4ce802e7b0 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:21.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.514+0000 7f4cf7805640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce80caf90 con 0x7f4ce802e7b0 2024-03-29T11:20:21.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.514+0000 7f4cf7805640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce802f740 con 0x7f4ce802b0f0 2024-03-29T11:20:21.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.514+0000 7f4cf7805640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce802d3c0 con 0x7f4ce8037220 2024-03-29T11:20:21.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.514+0000 7f4cf7805640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:21.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.514+0000 7f4cf7805640 10 monclient(hunting): authenticate will time out at 881.810360s 2024-03-29T11:20:21.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.515+0000 7f4cf4da0640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f4ce8037220 0x7f4ce80cbb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:37159/0 (socket says 172.21.15.67:37159) 2024-03-29T11:20:21.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.515+0000 7f4cf4da0640 1 -- 172.21.15.67:0/512029904 learned_addr learned my addr 172.21.15.67:0/512029904 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:21.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.515+0000 7f4ce5ffb640 10 client.?.objecter ms_handle_connect 0x7f4ce8037220 2024-03-29T11:20:21.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.515+0000 7f4ce5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:21.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 client.?.objecter ms_handle_connect 0x7f4ce802e7b0 2024-03-29T11:20:21.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:21.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785879212 0 0) 0x7f4ce802d3c0 con 0x7f4ce8037220 2024-03-29T11:20:21.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:21.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:21.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:21.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:20:21.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd4004eb0 con 0x7f4ce8037220 2024-03-29T11:20:21.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 client.?.objecter ms_handle_connect 0x7f4ce802b0f0 2024-03-29T11:20:21.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:21.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388575256 0 0) 0x7f4ce802f740 con 0x7f4ce802b0f0 2024-03-29T11:20:21.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:21.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:21.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:21.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:20:21.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce802d3c0 con 0x7f4ce802b0f0 2024-03-29T11:20:21.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230997164 0 0) 0x7f4ce80caf90 con 0x7f4ce802e7b0 2024-03-29T11:20:21.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:21.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:21.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:21.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:20:21.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.516+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd40063c0 con 0x7f4ce802e7b0 2024-03-29T11:20:21.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.517+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 860612842 0 0) 0x7f4ce802d3c0 con 0x7f4ce802b0f0 2024-03-29T11:20:21.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.517+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cd4006810 con 0x7f4ce802b0f0 2024-03-29T11:20:21.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.517+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f4cec001810 con 0x7f4ce802b0f0 2024-03-29T11:20:21.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.517+0000 7f4ce5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:21.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.517+0000 7f4ce5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:21.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.517+0000 7f4ce5ffb640 10 monclient(hunting): dump: 2024-03-29T11:20:21.497 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:21.497 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:21.497 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:21.497 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:21.497 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:21.497 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:21.497 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:21.497 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:21.498 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:21.498 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:21.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.517+0000 7f4ce5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:21.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1307099075 0 0) 0x7f4cd4004eb0 con 0x7f4ce8037220 2024-03-29T11:20:21.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ce802d3c0 con 0x7f4ce8037220 2024-03-29T11:20:21.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4123694411 0 0) 0x7f4cd4006810 con 0x7f4ce802b0f0 2024-03-29T11:20:21.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 >> v1:172.21.15.67:6790/0 conn(0x7f4ce8037220 legacy=0x7f4ce80cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 >> v1:172.21.15.67:6789/0 conn(0x7f4ce802e7b0 legacy=0x7f4ce816ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 1 monclient: found mon.b 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce81651d0 con 0x7f4ce802b0f0 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cec001d00 con 0x7f4ce802b0f0 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f4cec003830 con 0x7f4ce802b0f0 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:21.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.518+0000 7f4ce5ffb640 10 monclient: dump: 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.519+0000 7f4ce77fe640 4 set_mon_vals no callback set 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.520+0000 7f4cf7805640 5 monclient: authenticate success, global_id 4252 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.520+0000 7f4cf7805640 10 monclient: _renew_subs 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.520+0000 7f4cf7805640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:21.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.520+0000 7f4cf7805640 1 -- 172.21.15.67:0/512029904 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce8164790 con 0x7f4ce802b0f0 2024-03-29T11:20:21.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.520+0000 7f4cf7805640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:21.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.520+0000 7f4cf7805640 10 monclient: _renew_subs 2024-03-29T11:20:21.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.520+0000 7f4cf7805640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:21.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.520+0000 7f4cf7805640 1 -- 172.21.15.67:0/512029904 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce816d180 con 0x7f4ce802b0f0 2024-03-29T11:20:21.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.520+0000 7f4cf7805640 1 librados: init done 2024-03-29T11:20:21.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.526+0000 7f4cf7805640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:21.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.526+0000 7f4cf7805640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:21.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.526+0000 7f4cf7805640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:21.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.526+0000 7f4cf7805640 1 -- 172.21.15.67:0/512029904 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce8004cd0 con 0x7f4ce802b0f0 2024-03-29T11:20:21.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.526+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f4cec019800 con 0x7f4ce802b0f0 2024-03-29T11:20:21.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.527+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f4cec083530 con 0x7f4ce802b0f0 2024-03-29T11:20:21.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.527+0000 7f4ce5ffb640 10 client.4252.objecter ms_dispatch 0x7f4cf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:21.504 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.527+0000 7f4ce5ffb640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:21.504 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.527+0000 7f4ce5ffb640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:21.504 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.527+0000 7f4ce5ffb640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-29T11:20:21.504 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.527+0000 7f4cf4da0640 10 monclient: get_auth_request con 0x7f4cd406e320 auth_method 0 2024-03-29T11:20:21.504 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.528+0000 7f4ce5ffb640 10 client.4252.objecter ms_handle_connect 0x7f4cd406e320 2024-03-29T11:20:21.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.536+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4cec04ec40 con 0x7f4ce802b0f0 2024-03-29T11:20:21.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.536+0000 7f4ce5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:21.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.536+0000 7f4ce5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:21.840 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.863+0000 7f4cf7805640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:21.840 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.863+0000 7f4cf7805640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:21.840 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.863+0000 7f4cf7805640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:21.840 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.863+0000 7f4cf7805640 1 -- 172.21.15.67:0/512029904 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ce8039590 con 0x7f4ce802b0f0 2024-03-29T11:20:21.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.865+0000 7f4ce5ffb640 1 -- 172.21.15.67:0/512029904 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f4cec003690 con 0x7f4ce802b0f0 2024-03-29T11:20:21.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.865+0000 7f4ce5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:21.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.865+0000 7f4ce5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:21.842 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:20:21.842 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:21.845 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.868+0000 7f4cf7805640 10 librados: watch_flush enter 2024-03-29T11:20:21.846 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.869+0000 7f4cf7805640 10 librados: watch_flush exit 2024-03-29T11:20:21.846 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.869+0000 7f4cf7805640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-29T11:20:21.846 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.869+0000 7f4cf7805640 10 client.4252.objecter successfully canceled tick 2024-03-29T11:20:21.846 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.869+0000 7f4cf7805640 1 -- 172.21.15.67:0/512029904 >> v1:172.21.15.67:6816/404787197 conn(0x7f4cd406e320 legacy=0x7f4cd40707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:21.846 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.869+0000 7f4cf7805640 10 monclient: shutdown 2024-03-29T11:20:21.846 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.869+0000 7f4cf7805640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:21.846 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.869+0000 7f4cf7805640 1 -- 172.21.15.67:0/512029904 >> v1:172.21.15.82:6789/0 conn(0x7f4ce802b0f0 legacy=0x7f4ce80cb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:21.846 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.869+0000 7f4cf7805640 1 -- 172.21.15.67:0/512029904 shutdown_connections 2024-03-29T11:20:21.846 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.869+0000 7f4cf7805640 1 -- 172.21.15.67:0/512029904 >> 172.21.15.67:0/512029904 conn(0x7f4ce803a4b0 msgr2=0x7f4ce8038d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:21.846 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.870+0000 7f4cf7805640 1 -- 172.21.15.67:0/512029904 shutdown_connections 2024-03-29T11:20:21.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.870+0000 7f4cf7805640 1 -- 172.21.15.67:0/512029904 wait complete. 2024-03-29T11:20:21.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:21.870+0000 7f4cf7805640 1 librados: shutdown 2024-03-29T11:20:21.865 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:28.673 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:28.830 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.854+0000 7f9fe9831640 10 monclient: get_monmap_and_config 2024-03-29T11:20:28.830 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.854+0000 7f9fe9831640 10 monclient: build_initial_monmap 2024-03-29T11:20:28.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.854+0000 7f9fe9831640 10 monclient: monmap: 2024-03-29T11:20:28.831 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:28.831 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:28.831 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:20:28.854901+0000 2024-03-29T11:20:28.831 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:20:28.854901+0000 2024-03-29T11:20:28.831 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:28.831 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:28.831 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:28.831 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:20:28.831 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:20:28.831 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:28.836 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.855+0000 7f9fe9831640 1 Processor -- start 2024-03-29T11:20:28.836 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.855+0000 7f9fe9831640 1 -- start start 2024-03-29T11:20:28.836 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.855+0000 7f9fe9831640 10 monclient: init 2024-03-29T11:20:28.836 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.856+0000 7f9fe9831640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:28.836 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.856+0000 7f9fe9831640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:28.836 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.856+0000 7f9fe9831640 10 monclient(hunting): picked mon.noname-c con 0x7f9fe40b1f60 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:28.836 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.856+0000 7f9fe9831640 10 monclient(hunting): picked mon.noname-b con 0x7f9fe4071eb0 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:28.837 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.856+0000 7f9fe9831640 10 monclient(hunting): picked mon.noname-a con 0x7f9fe4143d10 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:28.837 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.856+0000 7f9fe9831640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe406ef80 con 0x7f9fe4143d10 2024-03-29T11:20:28.837 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.856+0000 7f9fe9831640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe406f260 con 0x7f9fe40b1f60 2024-03-29T11:20:28.837 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.856+0000 7f9fe9831640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe40b40c0 con 0x7f9fe4071eb0 2024-03-29T11:20:28.837 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.856+0000 7f9fe9831640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:28.837 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.856+0000 7f9fe9831640 10 monclient(hunting): authenticate will time out at 889.152848s 2024-03-29T11:20:28.837 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.859+0000 7f9fe27fc640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f9fe4071eb0 0x7f9fe4146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:55483/0 (socket says 172.21.15.67:55483) 2024-03-29T11:20:28.837 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.859+0000 7f9fe27fc640 1 -- 172.21.15.67:0/527698424 learned_addr learned my addr 172.21.15.67:0/527698424 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:28.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.861+0000 7f9fe1ffb640 1 -- 172.21.15.67:0/527698424 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707835038 0 0) 0x7f9fe40b40c0 con 0x7f9fe4071eb0 2024-03-29T11:20:28.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.861+0000 7f9fe1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:28.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.861+0000 7f9fe1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:28.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.861+0000 7f9fe1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:28.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.861+0000 7f9fe1ffb640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:20:28.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.862+0000 7f9fe1ffb640 1 -- 172.21.15.67:0/527698424 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fcc003520 con 0x7f9fe4071eb0 2024-03-29T11:20:28.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.862+0000 7f9fe1ffb640 1 -- 172.21.15.67:0/527698424 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233275543 0 0) 0x7f9fe406ef80 con 0x7f9fe4143d10 2024-03-29T11:20:28.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.862+0000 7f9fe1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:28.840 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.862+0000 7f9fe1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:28.840 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.862+0000 7f9fe1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:28.840 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.862+0000 7f9fe1ffb640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:20:28.840 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.862+0000 7f9fe1ffb640 1 -- 172.21.15.67:0/527698424 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe40b40c0 con 0x7f9fe4143d10 2024-03-29T11:20:28.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.863+0000 7f9fe1ffb640 1 -- 172.21.15.67:0/527698424 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519146074 0 0) 0x7f9fe406f260 con 0x7f9fe40b1f60 2024-03-29T11:20:28.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.863+0000 7f9fe1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:28.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.863+0000 7f9fe1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:28.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.863+0000 7f9fe1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:28.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.864+0000 7f9fe1ffb640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:20:28.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.864+0000 7f9fe1ffb640 1 -- 172.21.15.67:0/527698424 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe406ef80 con 0x7f9fe40b1f60 2024-03-29T11:20:28.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.864+0000 7f9fe1ffb640 1 -- 172.21.15.67:0/527698424 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3684665382 0 0) 0x7f9fcc003520 con 0x7f9fe4071eb0 2024-03-29T11:20:28.842 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.864+0000 7f9fe1ffb640 1 -- 172.21.15.67:0/527698424 >> v1:172.21.15.82:6789/0 conn(0x7f9fe40b1f60 legacy=0x7f9fe4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.842 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.864+0000 7f9fe1ffb640 1 -- 172.21.15.67:0/527698424 >> v1:172.21.15.67:6789/0 conn(0x7f9fe4143d10 legacy=0x7f9fe4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.866+0000 7f9fe1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:28.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.866+0000 7f9fe1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:20:28.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.866+0000 7f9fe1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:28.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.866+0000 7f9fe1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:20:28.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.866+0000 7f9fe1ffb640 1 -- 172.21.15.67:0/527698424 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe4155d70 con 0x7f9fe4071eb0 2024-03-29T11:20:28.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.866+0000 7f9fe1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:28.846 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.866+0000 7f9fe1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:28.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.866+0000 7f9fe1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:28.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.866+0000 7f9fe1ffb640 1 -- 172.21.15.67:0/527698424 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f9fd8001610 con 0x7f9fe4071eb0 2024-03-29T11:20:28.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.866+0000 7f9fe1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:28.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.866+0000 7f9fe1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:28.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.866+0000 7f9fe1ffb640 10 monclient: dump: 2024-03-29T11:20:28.847 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:28.847 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:28.847 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:28.847 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:28.847 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:28.847 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:28.847 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:28.848 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:28.848 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:28.848 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:28.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.866+0000 7f9fe9831640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:20:28.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.866+0000 7f9fe9831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:28.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.867+0000 7f9fe1ffb640 1 -- 172.21.15.67:0/527698424 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fd8001770 con 0x7f9fe4071eb0 2024-03-29T11:20:28.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.867+0000 7f9fe1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:28.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.867+0000 7f9fe1ffb640 1 -- 172.21.15.67:0/527698424 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f9fd8001b80 con 0x7f9fe4071eb0 2024-03-29T11:20:28.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.867+0000 7f9fe1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:28.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.867+0000 7f9fe1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:28.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.867+0000 7f9fe1ffb640 10 monclient: dump: 2024-03-29T11:20:28.848 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:28.849 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:28.849 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:28.849 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:28.849 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:28.849 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:28.849 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:28.849 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:28.849 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:28.849 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:28.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.867+0000 7f9fe9831640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:28.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.867+0000 7f9fe9831640 4 set_mon_vals no callback set 2024-03-29T11:20:28.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.868+0000 7f9fe9831640 10 monclient: shutdown 2024-03-29T11:20:28.850 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.868+0000 7f9fe9831640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:28.850 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.868+0000 7f9fe9831640 1 -- 172.21.15.67:0/527698424 >> v1:172.21.15.67:6790/0 conn(0x7f9fe4071eb0 legacy=0x7f9fe4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.850 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.872+0000 7f9fe9831640 1 -- 172.21.15.67:0/527698424 shutdown_connections 2024-03-29T11:20:28.850 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.872+0000 7f9fe9831640 1 -- 172.21.15.67:0/527698424 >> 172.21.15.67:0/527698424 conn(0x7f9fe4066a00 msgr2=0x7f9fe4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:28.850 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.874+0000 7f9fe9831640 1 -- 172.21.15.67:0/527698424 shutdown_connections 2024-03-29T11:20:28.850 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.874+0000 7f9fe9831640 1 -- 172.21.15.67:0/527698424 wait complete. 2024-03-29T11:20:28.852 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.874+0000 7f9fe9831640 10 monclient: build_initial_monmap 2024-03-29T11:20:28.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.874+0000 7f9fe9831640 10 monclient: monmap: 2024-03-29T11:20:28.853 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:28.853 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:28.853 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:20:28.853 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:20:28.853 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:28.853 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:28.853 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:28.853 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:20:28.853 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:20:28.853 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:28.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.875+0000 7f9fe9831640 1 librados: starting msgr at 2024-03-29T11:20:28.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.875+0000 7f9fe9831640 1 librados: starting objecter 2024-03-29T11:20:28.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.875+0000 7f9fe9831640 1 Processor -- start 2024-03-29T11:20:28.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.875+0000 7f9fe9831640 1 -- start start 2024-03-29T11:20:28.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.875+0000 7f9fe9831640 1 librados: setting wanted keys 2024-03-29T11:20:28.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.875+0000 7f9fe9831640 1 librados: calling monclient init 2024-03-29T11:20:28.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.875+0000 7f9fe9831640 10 monclient: init 2024-03-29T11:20:28.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.876+0000 7f9fe9831640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:28.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.876+0000 7f9fe9831640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:28.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.876+0000 7f9fe9831640 10 monclient(hunting): picked mon.noname-a con 0x7f9fe40b1f60 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:28.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.876+0000 7f9fe9831640 10 monclient(hunting): picked mon.noname-b con 0x7f9fe4143d10 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:28.857 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.880+0000 7f9fe9831640 10 monclient(hunting): picked mon.noname-c con 0x7f9fe4136a40 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:28.857 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.880+0000 7f9fe9831640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe41543f0 con 0x7f9fe40b1f60 2024-03-29T11:20:28.857 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.880+0000 7f9fe9831640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe41546d0 con 0x7f9fe4143d10 2024-03-29T11:20:28.857 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.880+0000 7f9fe9831640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe41549b0 con 0x7f9fe4136a40 2024-03-29T11:20:28.857 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.880+0000 7f9fe9831640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:28.857 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.880+0000 7f9fe9831640 10 monclient(hunting): authenticate will time out at 889.176373s 2024-03-29T11:20:28.857 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.880+0000 7f9fe2ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9fe40b1f60 0x7f9fe41423a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35595/0 (socket says 172.21.15.67:35595) 2024-03-29T11:20:28.857 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.880+0000 7f9fe2ffd640 1 -- 172.21.15.67:0/2479697917 learned_addr learned my addr 172.21.15.67:0/2479697917 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:28.858 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.880+0000 7f9fcbfff640 10 client.?.objecter ms_handle_connect 0x7f9fe40b1f60 2024-03-29T11:20:28.858 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.880+0000 7f9fcbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:28.860 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.883+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381824002 0 0) 0x7f9fe41543f0 con 0x7f9fe40b1f60 2024-03-29T11:20:28.860 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.883+0000 7f9fcbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:28.860 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.883+0000 7f9fcbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:28.861 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.883+0000 7f9fcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:28.861 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.883+0000 7f9fcbfff640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:20:28.861 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.883+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd0004eb0 con 0x7f9fe40b1f60 2024-03-29T11:20:28.861 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.883+0000 7f9fcbfff640 10 client.?.objecter ms_handle_connect 0x7f9fe4143d10 2024-03-29T11:20:28.861 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.883+0000 7f9fcbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:28.861 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.883+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 445416333 0 0) 0x7f9fd0004eb0 con 0x7f9fe40b1f60 2024-03-29T11:20:28.861 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.884+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fd0005300 con 0x7f9fe40b1f60 2024-03-29T11:20:28.861 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.884+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f9fdc001d70 con 0x7f9fe40b1f60 2024-03-29T11:20:28.861 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.884+0000 7f9fcbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:28.861 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.884+0000 7f9fcbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:28.861 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.884+0000 7f9fcbfff640 10 monclient(hunting): dump: 2024-03-29T11:20:28.861 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:28.862 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:28.862 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:28.862 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:28.862 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:28.862 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:28.862 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:28.862 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:28.862 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:28.862 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:28.862 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.884+0000 7f9fcbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:28.862 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.884+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459896128 0 0) 0x7f9fe41546d0 con 0x7f9fe4143d10 2024-03-29T11:20:28.863 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.884+0000 7f9fcbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:28.863 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.884+0000 7f9fcbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:28.863 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.884+0000 7f9fcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:28.863 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.884+0000 7f9fcbfff640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:20:28.863 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.884+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd0005850 con 0x7f9fe4143d10 2024-03-29T11:20:28.867 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.884+0000 7f9fcbfff640 10 client.?.objecter ms_handle_connect 0x7f9fe4136a40 2024-03-29T11:20:28.867 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.887+0000 7f9fcbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:28.867 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.887+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4157099385 0 0) 0x7f9fd0005300 con 0x7f9fe40b1f60 2024-03-29T11:20:28.867 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 >> v1:172.21.15.67:6790/0 conn(0x7f9fe4136a40 legacy=0x7f9fe41417a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.867 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 >> v1:172.21.15.82:6789/0 conn(0x7f9fe4143d10 legacy=0x7f9fe4141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.868 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:28.868 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 1 monclient: found mon.a 2024-03-29T11:20:28.868 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:28.868 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:20:28.868 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe41550b0 con 0x7f9fe40b1f60 2024-03-29T11:20:28.868 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 10 monclient: _finish_auth 0 2024-03-29T11:20:28.868 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:28.868 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:28.868 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fdc002030 con 0x7f9fe40b1f60 2024-03-29T11:20:28.868 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:28.868 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f9fdc001d70 con 0x7f9fe40b1f60 2024-03-29T11:20:28.869 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:28.869 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:28.869 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fcbfff640 10 monclient: dump: 2024-03-29T11:20:28.869 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:28.869 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:28.869 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:28.869 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:28.869 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:28.869 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:28.869 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:28.869 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:28.869 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:28.870 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:28.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.888+0000 7f9fe17fa640 4 set_mon_vals no callback set 2024-03-29T11:20:28.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.889+0000 7f9fe9831640 5 monclient: authenticate success, global_id 4272 2024-03-29T11:20:28.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.889+0000 7f9fe9831640 10 monclient: _renew_subs 2024-03-29T11:20:28.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.889+0000 7f9fe9831640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:20:28.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.889+0000 7f9fe9831640 1 -- 172.21.15.67:0/2479697917 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe4153b50 con 0x7f9fe40b1f60 2024-03-29T11:20:28.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.889+0000 7f9fe9831640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:28.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.889+0000 7f9fe9831640 10 monclient: _renew_subs 2024-03-29T11:20:28.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.889+0000 7f9fe9831640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:20:28.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.889+0000 7f9fe9831640 1 -- 172.21.15.67:0/2479697917 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe4141eb0 con 0x7f9fe40b1f60 2024-03-29T11:20:28.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.889+0000 7f9fe9831640 1 librados: init done 2024-03-29T11:20:28.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.891+0000 7f9fe9831640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:28.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.891+0000 7f9fe9831640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:28.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.891+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f9fdc003cc0 con 0x7f9fe40b1f60 2024-03-29T11:20:28.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.892+0000 7f9fe9831640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:20:28.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.892+0000 7f9fe9831640 1 -- 172.21.15.67:0/2479697917 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe4153730 con 0x7f9fe40b1f60 2024-03-29T11:20:28.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.892+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f9fdc054860 con 0x7f9fe40b1f60 2024-03-29T11:20:28.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.892+0000 7f9fcbfff640 10 client.4272.objecter ms_dispatch 0x7f9fe4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:28.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.892+0000 7f9fcbfff640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:28.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.892+0000 7f9fcbfff640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:28.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.892+0000 7f9fcbfff640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-29T11:20:28.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.893+0000 7f9fe27fc640 10 monclient: get_auth_request con 0x7f9fd006e210 auth_method 0 2024-03-29T11:20:28.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.893+0000 7f9fcbfff640 10 client.4272.objecter ms_handle_connect 0x7f9fd006e210 2024-03-29T11:20:28.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.902+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9fdc050b00 con 0x7f9fe40b1f60 2024-03-29T11:20:28.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.902+0000 7f9fcbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:28.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:28.902+0000 7f9fcbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:29.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.235+0000 7f9fe9831640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:29.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.235+0000 7f9fe9831640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:29.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.235+0000 7f9fe9831640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:20:29.212 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.235+0000 7f9fe9831640 1 -- 172.21.15.67:0/2479697917 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fe4075f10 con 0x7f9fe40b1f60 2024-03-29T11:20:29.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.237+0000 7f9fcbfff640 1 -- 172.21.15.67:0/2479697917 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f9fdc050ca0 con 0x7f9fe40b1f60 2024-03-29T11:20:29.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.237+0000 7f9fcbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:29.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.237+0000 7f9fcbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:29.215 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:20:29.215 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:29.218 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.242+0000 7f9fe9831640 10 librados: watch_flush enter 2024-03-29T11:20:29.218 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.242+0000 7f9fe9831640 10 librados: watch_flush exit 2024-03-29T11:20:29.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.242+0000 7f9fe9831640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-29T11:20:29.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.242+0000 7f9fe9831640 10 client.4272.objecter successfully canceled tick 2024-03-29T11:20:29.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.242+0000 7f9fe9831640 1 -- 172.21.15.67:0/2479697917 >> v1:172.21.15.67:6816/404787197 conn(0x7f9fd006e210 legacy=0x7f9fd00706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:29.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.242+0000 7f9fe9831640 10 monclient: shutdown 2024-03-29T11:20:29.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.242+0000 7f9fe9831640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:29.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.242+0000 7f9fe9831640 1 -- 172.21.15.67:0/2479697917 >> v1:172.21.15.67:6789/0 conn(0x7f9fe40b1f60 legacy=0x7f9fe41423a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:29.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.244+0000 7f9fe9831640 1 -- 172.21.15.67:0/2479697917 shutdown_connections 2024-03-29T11:20:29.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.244+0000 7f9fe9831640 1 -- 172.21.15.67:0/2479697917 >> 172.21.15.67:0/2479697917 conn(0x7f9fe4066a00 msgr2=0x7f9fe41468f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:29.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.245+0000 7f9fe9831640 1 -- 172.21.15.67:0/2479697917 shutdown_connections 2024-03-29T11:20:29.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.245+0000 7f9fe9831640 1 -- 172.21.15.67:0/2479697917 wait complete. 2024-03-29T11:20:29.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:29.245+0000 7f9fe9831640 1 librados: shutdown 2024-03-29T11:20:29.239 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:36.046 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:36.210 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.234+0000 7f6eb8c96640 10 monclient: get_monmap_and_config 2024-03-29T11:20:36.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.234+0000 7f6eb8c96640 10 monclient: build_initial_monmap 2024-03-29T11:20:36.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.234+0000 7f6eb8c96640 10 monclient: monmap: 2024-03-29T11:20:36.211 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:36.211 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:36.211 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:20:36.235000+0000 2024-03-29T11:20:36.211 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:20:36.235000+0000 2024-03-29T11:20:36.211 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:36.211 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:36.211 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:36.211 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:20:36.211 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:20:36.212 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:36.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.235+0000 7f6eb8c96640 1 Processor -- start 2024-03-29T11:20:36.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.235+0000 7f6eb8c96640 1 -- start start 2024-03-29T11:20:36.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.235+0000 7f6eb8c96640 10 monclient: init 2024-03-29T11:20:36.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.236+0000 7f6eb8c96640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:36.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.236+0000 7f6eb8c96640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:36.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.236+0000 7f6eb8c96640 10 monclient(hunting): picked mon.noname-a con 0x7f6eb40b0770 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:36.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.236+0000 7f6eb8c96640 10 monclient(hunting): picked mon.noname-b con 0x7f6eb414a700 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:36.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.236+0000 7f6eb8c96640 10 monclient(hunting): picked mon.noname-c con 0x7f6eb40ac5a0 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:36.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.236+0000 7f6eb8c96640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb40aba70 con 0x7f6eb40b0770 2024-03-29T11:20:36.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.236+0000 7f6eb8c96640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb40aecb0 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.236+0000 7f6eb8c96640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb40af150 con 0x7f6eb414a700 2024-03-29T11:20:36.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.236+0000 7f6eb8c96640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:36.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.236+0000 7f6eb8c96640 10 monclient(hunting): authenticate will time out at 896.532172s 2024-03-29T11:20:36.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.236+0000 7f6eb1d9b640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f6eb414a700 0x7f6eb40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:56524/0 (socket says 172.21.15.67:56524) 2024-03-29T11:20:36.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.236+0000 7f6eb1d9b640 1 -- 172.21.15.67:0/660217795 learned_addr learned my addr 172.21.15.67:0/660217795 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:36.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.237+0000 7f6eb159a640 1 -- 172.21.15.67:0/660217795 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979596988 0 0) 0x7f6eb40af150 con 0x7f6eb414a700 2024-03-29T11:20:36.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.237+0000 7f6eb159a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:36.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.237+0000 7f6eb159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:36.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.237+0000 7f6eb159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:36.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.237+0000 7f6eb159a640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:20:36.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.237+0000 7f6eb159a640 1 -- 172.21.15.67:0/660217795 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e9c003520 con 0x7f6eb414a700 2024-03-29T11:20:36.216 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.238+0000 7f6eb159a640 1 -- 172.21.15.67:0/660217795 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1311040557 0 0) 0x7f6e9c003520 con 0x7f6eb414a700 2024-03-29T11:20:36.216 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.238+0000 7f6eb159a640 1 -- 172.21.15.67:0/660217795 >> v1:172.21.15.82:6789/0 conn(0x7f6eb40ac5a0 legacy=0x7f6eb4151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:36.218 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.240+0000 7f6eb159a640 1 -- 172.21.15.67:0/660217795 >> v1:172.21.15.67:6789/0 conn(0x7f6eb40b0770 legacy=0x7f6eb4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.218 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.240+0000 7f6eb159a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:36.218 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.240+0000 7f6eb159a640 1 monclient: found mon.noname-b 2024-03-29T11:20:36.218 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.240+0000 7f6eb159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:36.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.240+0000 7f6eb159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:20:36.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.240+0000 7f6eb159a640 1 -- 172.21.15.67:0/660217795 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb40ac9d0 con 0x7f6eb414a700 2024-03-29T11:20:36.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.241+0000 7f6eb159a640 10 monclient: _finish_auth 0 2024-03-29T11:20:36.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.241+0000 7f6eb159a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:36.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.241+0000 7f6eb159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:36.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.241+0000 7f6eb8c96640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:20:36.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.241+0000 7f6eb8c96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:36.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.241+0000 7f6eb159a640 1 -- 172.21.15.67:0/660217795 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6ea8001690 con 0x7f6eb414a700 2024-03-29T11:20:36.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.241+0000 7f6eb159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:36.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.241+0000 7f6eb159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:36.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.241+0000 7f6eb159a640 10 monclient: dump: 2024-03-29T11:20:36.220 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:36.220 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:36.220 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:36.220 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:36.220 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:36.220 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:36.220 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:36.220 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:36.220 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:36.220 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:36.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.241+0000 7f6eb8c96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:36.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.241+0000 7f6eb159a640 1 -- 172.21.15.67:0/660217795 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ea8001920 con 0x7f6eb414a700 2024-03-29T11:20:36.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.241+0000 7f6eb159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:36.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.241+0000 7f6eb8c96640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:36.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.241+0000 7f6eb8c96640 4 set_mon_vals no callback set 2024-03-29T11:20:36.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.242+0000 7f6eb8c96640 10 monclient: shutdown 2024-03-29T11:20:36.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.242+0000 7f6eb8c96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:36.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.242+0000 7f6eb8c96640 1 -- 172.21.15.67:0/660217795 >> v1:172.21.15.67:6790/0 conn(0x7f6eb414a700 legacy=0x7f6eb40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.243+0000 7f6eb8c96640 1 -- 172.21.15.67:0/660217795 shutdown_connections 2024-03-29T11:20:36.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.243+0000 7f6eb8c96640 1 -- 172.21.15.67:0/660217795 >> 172.21.15.67:0/660217795 conn(0x7f6eb406b660 msgr2=0x7f6eb406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:36.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.243+0000 7f6eb8c96640 1 -- 172.21.15.67:0/660217795 shutdown_connections 2024-03-29T11:20:36.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.243+0000 7f6eb8c96640 1 -- 172.21.15.67:0/660217795 wait complete. 2024-03-29T11:20:36.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.245+0000 7f6eb8c96640 10 monclient: build_initial_monmap 2024-03-29T11:20:36.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.245+0000 7f6eb8c96640 10 monclient: monmap: 2024-03-29T11:20:36.224 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:36.224 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:36.224 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:20:36.224 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:20:36.224 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:36.224 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:36.224 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:36.224 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:20:36.224 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:20:36.224 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:36.224 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.245+0000 7f6eb8c96640 1 librados: starting msgr at 2024-03-29T11:20:36.224 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.245+0000 7f6eb8c96640 1 librados: starting objecter 2024-03-29T11:20:36.224 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.246+0000 7f6eb8c96640 1 Processor -- start 2024-03-29T11:20:36.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.246+0000 7f6eb8c96640 1 -- start start 2024-03-29T11:20:36.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.246+0000 7f6eb8c96640 1 librados: setting wanted keys 2024-03-29T11:20:36.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.246+0000 7f6eb8c96640 1 librados: calling monclient init 2024-03-29T11:20:36.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.246+0000 7f6eb8c96640 10 monclient: init 2024-03-29T11:20:36.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.247+0000 7f6eb8c96640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:36.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.247+0000 7f6eb8c96640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:36.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.247+0000 7f6eb8c96640 10 monclient(hunting): picked mon.noname-a con 0x7f6eb40ac5a0 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:36.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.247+0000 7f6eb8c96640 10 monclient(hunting): picked mon.noname-c con 0x7f6eb40b0770 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:36.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.247+0000 7f6eb8c96640 10 monclient(hunting): picked mon.noname-b con 0x7f6eb414a700 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:36.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.247+0000 7f6eb8c96640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb4126990 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.247+0000 7f6eb8c96640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb4128400 con 0x7f6eb414a700 2024-03-29T11:20:36.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.247+0000 7f6eb8c96640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb4128670 con 0x7f6eb40b0770 2024-03-29T11:20:36.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.247+0000 7f6eb8c96640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:36.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.247+0000 7f6eb8c96640 10 monclient(hunting): authenticate will time out at 896.543781s 2024-03-29T11:20:36.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.248+0000 7f6eb1d9b640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f6eb40b0770 0x7f6eb4145110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:56892/0 (socket says 172.21.15.67:56892) 2024-03-29T11:20:36.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.248+0000 7f6eb1d9b640 1 -- 172.21.15.67:0/2954485602 learned_addr learned my addr 172.21.15.67:0/2954485602 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:36.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 10 client.?.objecter ms_handle_connect 0x7f6eb40b0770 2024-03-29T11:20:36.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:36.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 10 client.?.objecter ms_handle_connect 0x7f6eb40ac5a0 2024-03-29T11:20:36.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:36.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480762832 0 0) 0x7f6eb4128670 con 0x7f6eb40b0770 2024-03-29T11:20:36.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:36.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:36.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:36.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:20:36.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e94004eb0 con 0x7f6eb40b0770 2024-03-29T11:20:36.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880745286 0 0) 0x7f6eb4126990 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:36.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:36.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:36.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:20:36.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.249+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eb4128670 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.250+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1026471458 0 0) 0x7f6eb4128670 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.230 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.250+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6eb4126990 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.230 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.250+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6ea4001bb0 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.230 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.250+0000 7f6ea2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:36.230 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.250+0000 7f6ea2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:36.230 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.250+0000 7f6ea2ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:36.230 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:36.230 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:36.230 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:36.230 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:36.230 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:36.230 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:36.231 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:36.231 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:36.231 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:36.231 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:36.231 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.250+0000 7f6ea2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:36.231 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.251+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1675859445 0 0) 0x7f6eb4126990 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.231 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.251+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 >> v1:172.21.15.67:6790/0 conn(0x7f6eb40b0770 legacy=0x7f6eb4145110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.231 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.251+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 >> v1:172.21.15.82:6789/0 conn(0x7f6eb414a700 legacy=0x7f6eb41dba50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:36.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.252+0000 7f6ea2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:36.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.252+0000 7f6ea2ffd640 1 monclient: found mon.a 2024-03-29T11:20:36.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.252+0000 7f6ea2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:36.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.252+0000 7f6ea2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:20:36.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.252+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb4127570 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6ea2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:36.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6ea2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:36.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6ea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:36.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6eb8c96640 5 monclient: authenticate success, global_id 4278 2024-03-29T11:20:36.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6eb8c96640 10 monclient: _renew_subs 2024-03-29T11:20:36.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6eb8c96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:20:36.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6eb8c96640 1 -- 172.21.15.67:0/2954485602 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb4127020 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6eb8c96640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:36.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6eb8c96640 10 monclient: _renew_subs 2024-03-29T11:20:36.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6eb8c96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:20:36.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6eb8c96640 1 -- 172.21.15.67:0/2954485602 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb4145a00 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6eb8c96640 1 librados: init done 2024-03-29T11:20:36.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ea4003990 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6ea2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:36.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6ea4002100 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.253+0000 7f6eb0d99640 4 set_mon_vals no callback set 2024-03-29T11:20:36.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.254+0000 7f6ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:36.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.255+0000 7f6ea2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:36.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.255+0000 7f6ea2ffd640 10 monclient: dump: 2024-03-29T11:20:36.237 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:36.237 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:36.237 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:36.237 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:36.237 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:36.238 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:36.238 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:36.238 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:36.238 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:36.238 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:36.238 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.255+0000 7f6ea0ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:36.238 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.255+0000 7f6ea0ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:36.238 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.255+0000 7f6ea0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:20:36.238 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.255+0000 7f6ea0ff9640 1 -- 172.21.15.67:0/2954485602 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb40ab560 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.238 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.257+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f6ea4003ca0 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.238 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.258+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f6ea4083b10 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.238 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.258+0000 7f6ea2ffd640 10 client.4278.objecter ms_dispatch 0x7f6eb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:36.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.258+0000 7f6ea2ffd640 3 client.4278.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:36.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.258+0000 7f6ea2ffd640 3 client.4278.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:36.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.258+0000 7f6ea2ffd640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-29T11:20:36.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.258+0000 7f6eb1d9b640 10 monclient: get_auth_request con 0x7f6e9406dda0 auth_method 0 2024-03-29T11:20:36.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.259+0000 7f6ea2ffd640 10 client.4278.objecter ms_handle_connect 0x7f6e9406dda0 2024-03-29T11:20:36.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.267+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ea404f3f0 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.267+0000 7f6ea2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:36.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.267+0000 7f6ea2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:36.644 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.668+0000 7f6ea0ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:36.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.668+0000 7f6ea0ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:36.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.668+0000 7f6ea0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:20:36.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.668+0000 7f6ea0ff9640 1 -- 172.21.15.67:0/2954485602 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6eb4127370 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.669+0000 7f6ea2ffd640 1 -- 172.21.15.67:0/2954485602 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f6ea4051320 con 0x7f6eb40ac5a0 2024-03-29T11:20:36.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.669+0000 7f6ea2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:36.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.669+0000 7f6ea2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:36.646 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:20:36.646 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:36.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.673+0000 7f6ea0ff9640 10 librados: watch_flush enter 2024-03-29T11:20:36.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.673+0000 7f6ea0ff9640 10 librados: watch_flush exit 2024-03-29T11:20:36.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.673+0000 7f6ea0ff9640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-29T11:20:36.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.673+0000 7f6ea0ff9640 10 client.4278.objecter successfully canceled tick 2024-03-29T11:20:36.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.673+0000 7f6ea0ff9640 1 -- 172.21.15.67:0/2954485602 >> v1:172.21.15.67:6816/404787197 conn(0x7f6e9406dda0 legacy=0x7f6e94070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.673+0000 7f6ea0ff9640 10 monclient: shutdown 2024-03-29T11:20:36.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.673+0000 7f6ea0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:36.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.673+0000 7f6ea0ff9640 1 -- 172.21.15.67:0/2954485602 >> v1:172.21.15.67:6789/0 conn(0x7f6eb40ac5a0 legacy=0x7f6eb4126280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.674+0000 7f6ea0ff9640 1 -- 172.21.15.67:0/2954485602 shutdown_connections 2024-03-29T11:20:36.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.674+0000 7f6ea0ff9640 1 -- 172.21.15.67:0/2954485602 >> 172.21.15.67:0/2954485602 conn(0x7f6eb406b660 msgr2=0x7f6eb40b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:36.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.674+0000 7f6ea0ff9640 1 -- 172.21.15.67:0/2954485602 shutdown_connections 2024-03-29T11:20:36.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.675+0000 7f6ea0ff9640 1 -- 172.21.15.67:0/2954485602 wait complete. 2024-03-29T11:20:36.652 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:36.675+0000 7f6ea0ff9640 1 librados: shutdown 2024-03-29T11:20:36.669 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:43.476 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:43.626 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.648+0000 7f7f2e3f6640 10 monclient: get_monmap_and_config 2024-03-29T11:20:43.626 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.648+0000 7f7f2e3f6640 10 monclient: build_initial_monmap 2024-03-29T11:20:43.626 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.648+0000 7f7f2e3f6640 10 monclient: monmap: 2024-03-29T11:20:43.626 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:43.626 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:43.626 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:20:43.650349+0000 2024-03-29T11:20:43.626 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:20:43.650349+0000 2024-03-29T11:20:43.626 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:43.626 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:43.627 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:43.627 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:20:43.627 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:20:43.627 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:43.627 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.649+0000 7f7f2e3f6640 1 Processor -- start 2024-03-29T11:20:43.627 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.649+0000 7f7f2e3f6640 1 -- start start 2024-03-29T11:20:43.628 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.649+0000 7f7f2e3f6640 10 monclient: init 2024-03-29T11:20:43.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.650+0000 7f7f2e3f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:43.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.650+0000 7f7f2e3f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:43.637 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.650+0000 7f7f2e3f6640 10 monclient(hunting): picked mon.noname-c con 0x7f7f280b0770 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:43.637 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.650+0000 7f7f2e3f6640 10 monclient(hunting): picked mon.noname-a con 0x7f7f2814a700 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:43.637 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.650+0000 7f7f2e3f6640 10 monclient(hunting): picked mon.noname-b con 0x7f7f280ac5a0 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:43.637 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.650+0000 7f7f2e3f6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f280aba70 con 0x7f7f2814a700 2024-03-29T11:20:43.637 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.650+0000 7f7f2e3f6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f280aecb0 con 0x7f7f280b0770 2024-03-29T11:20:43.638 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.650+0000 7f7f2e3f6640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f280af150 con 0x7f7f280ac5a0 2024-03-29T11:20:43.638 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.650+0000 7f7f2e3f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:43.638 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.650+0000 7f7f2e3f6640 10 monclient(hunting): authenticate will time out at 903.947658s 2024-03-29T11:20:43.638 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.651+0000 7f7f27fff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f7f280b0770 0x7f7f28149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:47708/0 (socket says 172.21.15.67:47708) 2024-03-29T11:20:43.638 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.651+0000 7f7f27fff640 1 -- 172.21.15.67:0/879074314 learned_addr learned my addr 172.21.15.67:0/879074314 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:43.638 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.651+0000 7f7f26ffd640 1 -- 172.21.15.67:0/879074314 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010994111 0 0) 0x7f7f280aecb0 con 0x7f7f280b0770 2024-03-29T11:20:43.638 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.651+0000 7f7f26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:43.638 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.651+0000 7f7f26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:43.638 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.651+0000 7f7f26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:43.638 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.651+0000 7f7f26ffd640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:20:43.639 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.651+0000 7f7f26ffd640 1 -- 172.21.15.67:0/879074314 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f14003520 con 0x7f7f280b0770 2024-03-29T11:20:43.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.651+0000 7f7f26ffd640 1 -- 172.21.15.67:0/879074314 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801814688 0 0) 0x7f7f280aba70 con 0x7f7f2814a700 2024-03-29T11:20:43.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.651+0000 7f7f26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:43.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.651+0000 7f7f26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:43.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.651+0000 7f7f26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:43.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.651+0000 7f7f26ffd640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:20:43.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.651+0000 7f7f26ffd640 1 -- 172.21.15.67:0/879074314 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f280aecb0 con 0x7f7f2814a700 2024-03-29T11:20:43.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 1 -- 172.21.15.67:0/879074314 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1629427726 0 0) 0x7f7f14003520 con 0x7f7f280b0770 2024-03-29T11:20:43.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 1 -- 172.21.15.67:0/879074314 >> v1:172.21.15.67:6790/0 conn(0x7f7f280ac5a0 legacy=0x7f7f28151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 1 -- 172.21.15.67:0/879074314 >> v1:172.21.15.67:6789/0 conn(0x7f7f2814a700 legacy=0x7f7f280abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:43.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 1 monclient: found mon.noname-c 2024-03-29T11:20:43.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:43.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:20:43.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 1 -- 172.21.15.67:0/879074314 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f280ac9d0 con 0x7f7f280b0770 2024-03-29T11:20:43.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:43.644 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:43.644 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:43.644 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 1 -- 172.21.15.67:0/879074314 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7f18001a30 con 0x7f7f280b0770 2024-03-29T11:20:43.644 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:43.644 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:43.644 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 10 monclient: dump: 2024-03-29T11:20:43.644 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:43.644 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:43.644 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:43.644 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:43.644 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:43.644 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:43.645 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:43.645 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:43.645 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:43.645 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:43.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f2e3f6640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:20:43.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f2e3f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:43.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 1 -- 172.21.15.67:0/879074314 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f18001b90 con 0x7f7f280b0770 2024-03-29T11:20:43.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:43.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 1 -- 172.21.15.67:0/879074314 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7f18001fa0 con 0x7f7f280b0770 2024-03-29T11:20:43.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:43.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:43.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f26ffd640 10 monclient: dump: 2024-03-29T11:20:43.646 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:43.646 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:43.646 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:43.646 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:43.646 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:43.646 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:43.646 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:43.646 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:43.646 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:43.646 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:43.646 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f2e3f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:43.646 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.652+0000 7f7f2e3f6640 4 set_mon_vals no callback set 2024-03-29T11:20:43.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.653+0000 7f7f2e3f6640 10 monclient: shutdown 2024-03-29T11:20:43.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.653+0000 7f7f2e3f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:43.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.653+0000 7f7f2e3f6640 1 -- 172.21.15.67:0/879074314 >> v1:172.21.15.82:6789/0 conn(0x7f7f280b0770 legacy=0x7f7f28149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.653+0000 7f7f2e3f6640 1 -- 172.21.15.67:0/879074314 shutdown_connections 2024-03-29T11:20:43.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.653+0000 7f7f2e3f6640 1 -- 172.21.15.67:0/879074314 >> 172.21.15.67:0/879074314 conn(0x7f7f2806b660 msgr2=0x7f7f2806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:43.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.653+0000 7f7f2e3f6640 1 -- 172.21.15.67:0/879074314 shutdown_connections 2024-03-29T11:20:43.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.653+0000 7f7f2e3f6640 1 -- 172.21.15.67:0/879074314 wait complete. 2024-03-29T11:20:43.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.654+0000 7f7f2e3f6640 10 monclient: build_initial_monmap 2024-03-29T11:20:43.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.654+0000 7f7f2e3f6640 10 monclient: monmap: 2024-03-29T11:20:43.647 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:43.647 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:43.647 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:20:43.648 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:20:43.648 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:43.648 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:43.648 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:43.648 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:20:43.648 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:20:43.648 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:43.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.655+0000 7f7f2e3f6640 1 librados: starting msgr at 2024-03-29T11:20:43.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.655+0000 7f7f2e3f6640 1 librados: starting objecter 2024-03-29T11:20:43.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.655+0000 7f7f2e3f6640 1 Processor -- start 2024-03-29T11:20:43.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.655+0000 7f7f2e3f6640 1 -- start start 2024-03-29T11:20:43.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.655+0000 7f7f2e3f6640 1 librados: setting wanted keys 2024-03-29T11:20:43.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.655+0000 7f7f2e3f6640 1 librados: calling monclient init 2024-03-29T11:20:43.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.655+0000 7f7f2e3f6640 10 monclient: init 2024-03-29T11:20:43.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.656+0000 7f7f2e3f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:43.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.656+0000 7f7f2e3f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:43.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.656+0000 7f7f2e3f6640 10 monclient(hunting): picked mon.noname-b con 0x7f7f280ac5a0 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:43.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.656+0000 7f7f2e3f6640 10 monclient(hunting): picked mon.noname-a con 0x7f7f280b0770 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:43.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.656+0000 7f7f2e3f6640 10 monclient(hunting): picked mon.noname-c con 0x7f7f2814a700 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:43.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.656+0000 7f7f2e3f6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2812a450 con 0x7f7f280b0770 2024-03-29T11:20:43.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.656+0000 7f7f2e3f6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f281292b0 con 0x7f7f280ac5a0 2024-03-29T11:20:43.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.656+0000 7f7f2e3f6640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f281d91f0 con 0x7f7f2814a700 2024-03-29T11:20:43.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.656+0000 7f7f2e3f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:43.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.656+0000 7f7f2e3f6640 10 monclient(hunting): authenticate will time out at 903.953362s 2024-03-29T11:20:43.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.656+0000 7f7f27fff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f7f280ac5a0 0x7f7f28128060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:47848/0 (socket says 172.21.15.67:47848) 2024-03-29T11:20:43.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.656+0000 7f7f27fff640 1 -- 172.21.15.67:0/3618237857 learned_addr learned my addr 172.21.15.67:0/3618237857 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:43.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.656+0000 7f7f24ff9640 10 client.?.objecter ms_handle_connect 0x7f7f280ac5a0 2024-03-29T11:20:43.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.656+0000 7f7f24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:43.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 10 client.?.objecter ms_handle_connect 0x7f7f280b0770 2024-03-29T11:20:43.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:43.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272001361 0 0) 0x7f7f281292b0 con 0x7f7f280ac5a0 2024-03-29T11:20:43.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:43.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:43.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:43.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:20:43.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f08004eb0 con 0x7f7f280ac5a0 2024-03-29T11:20:43.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 10 client.?.objecter ms_handle_connect 0x7f7f2814a700 2024-03-29T11:20:43.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:43.650 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973610234 0 0) 0x7f7f2812a450 con 0x7f7f280b0770 2024-03-29T11:20:43.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:43.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:43.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:43.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:20:43.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.657+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f080051d0 con 0x7f7f280b0770 2024-03-29T11:20:43.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996046787 0 0) 0x7f7f281d91f0 con 0x7f7f2814a700 2024-03-29T11:20:43.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:43.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:43.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:43.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:20:43.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f2812a450 con 0x7f7f2814a700 2024-03-29T11:20:43.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3139131962 0 0) 0x7f7f08004eb0 con 0x7f7f280ac5a0 2024-03-29T11:20:43.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f281d91f0 con 0x7f7f280ac5a0 2024-03-29T11:20:43.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7f18001ee0 con 0x7f7f280ac5a0 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:43.652 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2786382851 0 0) 0x7f7f080051d0 con 0x7f7f280b0770 2024-03-29T11:20:43.653 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f08005cd0 con 0x7f7f280b0770 2024-03-29T11:20:43.653 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7f20001850 con 0x7f7f280b0770 2024-03-29T11:20:43.653 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:43.653 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:43.653 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:43.653 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:43.653 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:43.653 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:43.653 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:43.653 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:43.653 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:43.653 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:43.653 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:43.654 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:43.654 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:43.654 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 300500930 0 0) 0x7f7f281d91f0 con 0x7f7f280ac5a0 2024-03-29T11:20:43.654 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 >> v1:172.21.15.67:6790/0 conn(0x7f7f2814a700 legacy=0x7f7f281e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.654 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 >> v1:172.21.15.67:6789/0 conn(0x7f7f280b0770 legacy=0x7f7f281db990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.654 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:43.654 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 1 monclient: found mon.b 2024-03-29T11:20:43.654 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:43.654 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:43.654 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f28128dd0 con 0x7f7f280ac5a0 2024-03-29T11:20:43.654 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:43.654 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:43.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:43.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f2e3f6640 5 monclient: authenticate success, global_id 4264 2024-03-29T11:20:43.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f2e3f6640 10 monclient: _renew_subs 2024-03-29T11:20:43.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f2e3f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:43.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f2e3f6640 1 -- 172.21.15.67:0/3618237857 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f28129080 con 0x7f7f280ac5a0 2024-03-29T11:20:43.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f2e3f6640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:43.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f2e3f6640 10 monclient: _renew_subs 2024-03-29T11:20:43.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f2e3f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:43.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f2e3f6640 1 -- 172.21.15.67:0/3618237857 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f281dc280 con 0x7f7f280ac5a0 2024-03-29T11:20:43.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.658+0000 7f7f2e3f6640 1 librados: init done 2024-03-29T11:20:43.658 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.659+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f18004e10 con 0x7f7f280ac5a0 2024-03-29T11:20:43.658 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.659+0000 7f7f24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:43.658 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.659+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7f18001ee0 con 0x7f7f280ac5a0 2024-03-29T11:20:43.658 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.659+0000 7f7f24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:43.658 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.660+0000 7f7f24ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:43.658 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.660+0000 7f7f24ff9640 10 monclient: dump: 2024-03-29T11:20:43.658 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:43.658 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:43.658 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:43.658 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:43.658 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:43.658 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:43.658 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:43.659 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:43.659 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:43.659 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:43.659 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.660+0000 7f7f267fc640 4 set_mon_vals no callback set 2024-03-29T11:20:43.659 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.660+0000 7f7f1e7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:43.659 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.661+0000 7f7f1e7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:43.659 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.661+0000 7f7f1e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:43.659 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.661+0000 7f7f1e7fc640 1 -- 172.21.15.67:0/3618237857 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f280ab560 con 0x7f7f280ac5a0 2024-03-29T11:20:43.659 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.662+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f7f1801adf0 con 0x7f7f280ac5a0 2024-03-29T11:20:43.659 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.662+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f7f18084a90 con 0x7f7f280ac5a0 2024-03-29T11:20:43.659 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.662+0000 7f7f24ff9640 10 client.4264.objecter ms_dispatch 0x7f7f28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:43.659 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.662+0000 7f7f24ff9640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:43.659 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.662+0000 7f7f24ff9640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:43.660 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.662+0000 7f7f24ff9640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-29T11:20:43.660 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.663+0000 7f7f277fe640 10 monclient: get_auth_request con 0x7f7f0806e5b0 auth_method 0 2024-03-29T11:20:43.660 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.663+0000 7f7f24ff9640 10 client.4264.objecter ms_handle_connect 0x7f7f0806e5b0 2024-03-29T11:20:43.660 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.669+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f180503d0 con 0x7f7f280ac5a0 2024-03-29T11:20:43.660 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.669+0000 7f7f24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:43.660 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:43.669+0000 7f7f24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:44.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.094+0000 7f7f1e7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:44.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.094+0000 7f7f1e7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:44.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.094+0000 7f7f1e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:20:44.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.094+0000 7f7f1e7fc640 1 -- 172.21.15.67:0/3618237857 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f280ab560 con 0x7f7f280ac5a0 2024-03-29T11:20:44.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.095+0000 7f7f24ff9640 1 -- 172.21.15.67:0/3618237857 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f7f180503d0 con 0x7f7f280ac5a0 2024-03-29T11:20:44.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.095+0000 7f7f24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:44.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.095+0000 7f7f24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:44.073 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:20:44.073 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:44.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.099+0000 7f7f1e7fc640 10 librados: watch_flush enter 2024-03-29T11:20:44.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.099+0000 7f7f1e7fc640 10 librados: watch_flush exit 2024-03-29T11:20:44.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.099+0000 7f7f1e7fc640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-29T11:20:44.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.099+0000 7f7f1e7fc640 10 client.4264.objecter successfully canceled tick 2024-03-29T11:20:44.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.099+0000 7f7f1e7fc640 1 -- 172.21.15.67:0/3618237857 >> v1:172.21.15.67:6816/404787197 conn(0x7f7f0806e5b0 legacy=0x7f7f08070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:44.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.100+0000 7f7f1e7fc640 10 monclient: shutdown 2024-03-29T11:20:44.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.100+0000 7f7f1e7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:44.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.100+0000 7f7f1e7fc640 1 -- 172.21.15.67:0/3618237857 >> v1:172.21.15.82:6789/0 conn(0x7f7f280ac5a0 legacy=0x7f7f28128060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:44.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.100+0000 7f7f1e7fc640 1 -- 172.21.15.67:0/3618237857 shutdown_connections 2024-03-29T11:20:44.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.100+0000 7f7f1e7fc640 1 -- 172.21.15.67:0/3618237857 >> 172.21.15.67:0/3618237857 conn(0x7f7f2806b660 msgr2=0x7f7f280b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:44.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.100+0000 7f7f1e7fc640 1 -- 172.21.15.67:0/3618237857 shutdown_connections 2024-03-29T11:20:44.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.100+0000 7f7f1e7fc640 1 -- 172.21.15.67:0/3618237857 wait complete. 2024-03-29T11:20:44.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:44.100+0000 7f7f1e7fc640 1 librados: shutdown 2024-03-29T11:20:44.093 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:50.900 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:51.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.074+0000 7f3fbe2ff640 10 monclient: get_monmap_and_config 2024-03-29T11:20:51.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.074+0000 7f3fbe2ff640 10 monclient: build_initial_monmap 2024-03-29T11:20:51.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.074+0000 7f3fbe2ff640 10 monclient: monmap: 2024-03-29T11:20:51.051 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:51.052 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:51.052 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:20:51.075979+0000 2024-03-29T11:20:51.052 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:20:51.075979+0000 2024-03-29T11:20:51.052 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:51.052 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:51.052 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:51.052 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:20:51.052 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:20:51.052 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:51.053 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.076+0000 7f3fbe2ff640 1 Processor -- start 2024-03-29T11:20:51.053 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.076+0000 7f3fbe2ff640 1 -- start start 2024-03-29T11:20:51.053 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.076+0000 7f3fbe2ff640 10 monclient: init 2024-03-29T11:20:51.056 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fbe2ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:51.056 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fbe2ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:51.056 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fbe2ff640 10 monclient(hunting): picked mon.noname-a con 0x7f3fb81422b0 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:51.056 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fbe2ff640 10 monclient(hunting): picked mon.noname-b con 0x7f3fb8142690 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:51.056 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fbe2ff640 10 monclient(hunting): picked mon.noname-c con 0x7f3fb80b17d0 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:51.056 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fbe2ff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb8142c80 con 0x7f3fb81422b0 2024-03-29T11:20:51.056 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fbe2ff640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb80aae40 con 0x7f3fb80b17d0 2024-03-29T11:20:51.056 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fbe2ff640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb813fbd0 con 0x7f3fb8142690 2024-03-29T11:20:51.056 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fbe2ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:51.056 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fbe2ff640 10 monclient(hunting): authenticate will time out at 911.374005s 2024-03-29T11:20:51.056 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fb77fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f3fb8142690 0x7f3fb80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:37640/0 (socket says 172.21.15.67:37640) 2024-03-29T11:20:51.056 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fb77fe640 1 -- 172.21.15.67:0/2926521347 learned_addr learned my addr 172.21.15.67:0/2926521347 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:51.057 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fb6ffd640 1 -- 172.21.15.67:0/2926521347 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469759327 0 0) 0x7f3fb813fbd0 con 0x7f3fb8142690 2024-03-29T11:20:51.057 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fb6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.057 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.057 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.057 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fb6ffd640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:20:51.057 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.077+0000 7f3fb6ffd640 1 -- 172.21.15.67:0/2926521347 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa0003520 con 0x7f3fb8142690 2024-03-29T11:20:51.057 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.078+0000 7f3fb6ffd640 1 -- 172.21.15.67:0/2926521347 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1187409582 0 0) 0x7f3fa0003520 con 0x7f3fb8142690 2024-03-29T11:20:51.057 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.078+0000 7f3fb6ffd640 1 -- 172.21.15.67:0/2926521347 >> v1:172.21.15.82:6789/0 conn(0x7f3fb80b17d0 legacy=0x7f3fb80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:51.057 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.078+0000 7f3fb6ffd640 1 -- 172.21.15.67:0/2926521347 >> v1:172.21.15.67:6789/0 conn(0x7f3fb81422b0 legacy=0x7f3fb813f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:51.061 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.079+0000 7f3fb6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:51.061 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.079+0000 7f3fb6ffd640 1 monclient: found mon.noname-b 2024-03-29T11:20:51.061 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.079+0000 7f3fb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:51.061 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.079+0000 7f3fb6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:20:51.061 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.079+0000 7f3fb6ffd640 1 -- 172.21.15.67:0/2926521347 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb8151940 con 0x7f3fb8142690 2024-03-29T11:20:51.061 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fb6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:51.061 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fb6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:51.061 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:51.061 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fb6ffd640 1 -- 172.21.15.67:0/2926521347 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f3fac001670 con 0x7f3fb8142690 2024-03-29T11:20:51.061 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.062 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fb6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:51.062 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fb6ffd640 10 monclient: dump: 2024-03-29T11:20:51.062 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:51.062 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:51.062 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:51.062 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:51.062 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.062 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:51.062 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:51.062 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:51.062 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:51.062 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:51.063 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fbe2ff640 5 monclient: authenticate success, global_id 4274 2024-03-29T11:20:51.063 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fbe2ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:51.063 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fb6ffd640 1 -- 172.21.15.67:0/2926521347 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fac0018d0 con 0x7f3fb8142690 2024-03-29T11:20:51.063 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:51.063 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fb6ffd640 1 -- 172.21.15.67:0/2926521347 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f3fac001ca0 con 0x7f3fb8142690 2024-03-29T11:20:51.063 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.063 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fb6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:51.063 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.080+0000 7f3fb6ffd640 10 monclient: dump: 2024-03-29T11:20:51.063 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:51.063 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:51.063 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:51.063 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:51.063 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.063 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:51.064 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:51.064 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:51.064 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:51.064 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:51.064 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.081+0000 7f3fbe2ff640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:51.064 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.081+0000 7f3fbe2ff640 4 set_mon_vals no callback set 2024-03-29T11:20:51.064 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.081+0000 7f3fbe2ff640 10 monclient: shutdown 2024-03-29T11:20:51.064 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.081+0000 7f3fbe2ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:51.064 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.081+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2926521347 >> v1:172.21.15.67:6790/0 conn(0x7f3fb8142690 legacy=0x7f3fb80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.064 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.082+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2926521347 shutdown_connections 2024-03-29T11:20:51.064 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.082+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2926521347 >> 172.21.15.67:0/2926521347 conn(0x7f3fb8066a00 msgr2=0x7f3fb8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:51.064 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.082+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2926521347 shutdown_connections 2024-03-29T11:20:51.064 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.082+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2926521347 wait complete. 2024-03-29T11:20:51.064 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.083+0000 7f3fbe2ff640 10 monclient: build_initial_monmap 2024-03-29T11:20:51.065 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.083+0000 7f3fbe2ff640 10 monclient: monmap: 2024-03-29T11:20:51.065 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:51.065 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:51.065 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:20:51.065 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:20:51.065 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:51.065 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:51.065 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:51.065 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:20:51.065 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:20:51.065 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:51.065 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.083+0000 7f3fbe2ff640 1 librados: starting msgr at 2024-03-29T11:20:51.065 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.083+0000 7f3fbe2ff640 1 librados: starting objecter 2024-03-29T11:20:51.065 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.083+0000 7f3fbe2ff640 1 Processor -- start 2024-03-29T11:20:51.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.084+0000 7f3fbe2ff640 1 -- start start 2024-03-29T11:20:51.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.084+0000 7f3fbe2ff640 1 librados: setting wanted keys 2024-03-29T11:20:51.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.084+0000 7f3fbe2ff640 1 librados: calling monclient init 2024-03-29T11:20:51.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.084+0000 7f3fbe2ff640 10 monclient: init 2024-03-29T11:20:51.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.085+0000 7f3fbe2ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:51.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.085+0000 7f3fbe2ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:51.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.085+0000 7f3fbe2ff640 10 monclient(hunting): picked mon.noname-a con 0x7f3fb80b17d0 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:51.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.085+0000 7f3fbe2ff640 10 monclient(hunting): picked mon.noname-b con 0x7f3fb81422b0 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:51.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.085+0000 7f3fbe2ff640 10 monclient(hunting): picked mon.noname-c con 0x7f3fb8142690 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:51.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.085+0000 7f3fbe2ff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb8132050 con 0x7f3fb80b17d0 2024-03-29T11:20:51.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.085+0000 7f3fbe2ff640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb812b640 con 0x7f3fb81422b0 2024-03-29T11:20:51.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.085+0000 7f3fbc89c640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f3fb8142690 0x7f3fb812af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:38068/0 (socket says 172.21.15.67:38068) 2024-03-29T11:20:51.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.085+0000 7f3fbc89c640 1 -- 172.21.15.67:0/2718145352 learned_addr learned my addr 172.21.15.67:0/2718145352 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:51.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.085+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2718145352 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb812ca80 con 0x7f3fb8142690 2024-03-29T11:20:51.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.085+0000 7f3fb4ff9640 10 client.?.objecter ms_handle_connect 0x7f3fb8142690 2024-03-29T11:20:51.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.085+0000 7f3fb4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fbe2ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:51.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800843529 0 0) 0x7f3fb812ca80 con 0x7f3fb8142690 2024-03-29T11:20:51.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fbe2ff640 10 monclient(hunting): authenticate will time out at 911.383122s 2024-03-29T11:20:51.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:20:51.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa4004eb0 con 0x7f3fb8142690 2024-03-29T11:20:51.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3609744101 0 0) 0x7f3fa4004eb0 con 0x7f3fb8142690 2024-03-29T11:20:51.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fb812ca80 con 0x7f3fb8142690 2024-03-29T11:20:51.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f3fb0001c60 con 0x7f3fb8142690 2024-03-29T11:20:51.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:51.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:51.071 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:51.071 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:51.071 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:51.071 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 10 client.?.objecter ms_handle_connect 0x7f3fb81422b0 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.086+0000 7f3fb4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.087+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989402447 0 0) 0x7f3fb812b640 con 0x7f3fb81422b0 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.087+0000 7f3fb4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.087+0000 7f3fb4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.087+0000 7f3fb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.087+0000 7f3fb4ff9640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:20:51.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.087+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa4004eb0 con 0x7f3fb81422b0 2024-03-29T11:20:51.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.087+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1819059736 0 0) 0x7f3fb812ca80 con 0x7f3fb8142690 2024-03-29T11:20:51.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.087+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 >> v1:172.21.15.82:6789/0 conn(0x7f3fb81422b0 legacy=0x7f3fb812a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.087+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 >> v1:172.21.15.67:6789/0 conn(0x7f3fb80b17d0 legacy=0x7f3fb81dba10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:51.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.088+0000 7f3fb4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:51.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.088+0000 7f3fb4ff9640 1 monclient: found mon.c 2024-03-29T11:20:51.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.088+0000 7f3fb4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:51.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.088+0000 7f3fb4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:51.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.088+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb812cd60 con 0x7f3fb8142690 2024-03-29T11:20:51.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.088+0000 7f3fb4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:51.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.088+0000 7f3fb4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:51.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.088+0000 7f3fb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:51.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fb0003970 con 0x7f3fb8142690 2024-03-29T11:20:51.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fb4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:51.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f3fb0003d80 con 0x7f3fb8142690 2024-03-29T11:20:51.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fb67fc640 4 set_mon_vals no callback set 2024-03-29T11:20:51.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fb4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:51.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fb4ff9640 10 monclient: dump: 2024-03-29T11:20:51.074 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:51.074 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:51.074 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:51.074 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:51.074 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.074 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:51.074 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:51.074 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:51.074 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:51.075 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:51.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fbe2ff640 5 monclient: authenticate success, global_id 4277 2024-03-29T11:20:51.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fbe2ff640 10 monclient: _renew_subs 2024-03-29T11:20:51.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fbe2ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:51.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2718145352 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb81324e0 con 0x7f3fb8142690 2024-03-29T11:20:51.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fbe2ff640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:51.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fbe2ff640 10 monclient: _renew_subs 2024-03-29T11:20:51.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fbe2ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:51.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.089+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2718145352 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb8131630 con 0x7f3fb8142690 2024-03-29T11:20:51.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.090+0000 7f3fbe2ff640 1 librados: init done 2024-03-29T11:20:51.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.091+0000 7f3fbe2ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.091+0000 7f3fbe2ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.091+0000 7f3fbe2ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:51.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.091+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2718145352 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb8075f10 con 0x7f3fb8142690 2024-03-29T11:20:51.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.092+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f3fb0003ee0 con 0x7f3fb8142690 2024-03-29T11:20:51.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.093+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f3fb0084e60 con 0x7f3fb8142690 2024-03-29T11:20:51.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.093+0000 7f3fb4ff9640 10 client.4277.objecter ms_dispatch 0x7f3fb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:51.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.093+0000 7f3fb4ff9640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:51.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.093+0000 7f3fb4ff9640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:51.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.093+0000 7f3fb4ff9640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-29T11:20:51.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.093+0000 7f3fb7fff640 10 monclient: get_auth_request con 0x7f3fa406dce0 auth_method 0 2024-03-29T11:20:51.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.094+0000 7f3fb4ff9640 10 client.4277.objecter ms_handle_connect 0x7f3fa406dce0 2024-03-29T11:20:51.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.100+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3fb0050740 con 0x7f3fb8142690 2024-03-29T11:20:51.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.100+0000 7f3fb4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.100+0000 7f3fb4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:51.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.407+0000 7f3fbe2ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:51.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.407+0000 7f3fbe2ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:51.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.407+0000 7f3fbe2ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:51.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.407+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2718145352 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fb80b2270 con 0x7f3fb8142690 2024-03-29T11:20:51.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.407+0000 7f3fb4ff9640 1 -- 172.21.15.67:0/2718145352 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f3fb0052670 con 0x7f3fb8142690 2024-03-29T11:20:51.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.408+0000 7f3fb4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:51.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.408+0000 7f3fb4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:51.387 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:20:51.387 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:51.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.411+0000 7f3fbe2ff640 10 librados: watch_flush enter 2024-03-29T11:20:51.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.411+0000 7f3fbe2ff640 10 librados: watch_flush exit 2024-03-29T11:20:51.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.411+0000 7f3fbe2ff640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-29T11:20:51.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.411+0000 7f3fbe2ff640 10 client.4277.objecter successfully canceled tick 2024-03-29T11:20:51.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.411+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2718145352 >> v1:172.21.15.67:6816/404787197 conn(0x7f3fa406dce0 legacy=0x7f3fa40701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.411+0000 7f3fbe2ff640 10 monclient: shutdown 2024-03-29T11:20:51.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.411+0000 7f3fbe2ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:51.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.411+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2718145352 >> v1:172.21.15.67:6790/0 conn(0x7f3fb8142690 legacy=0x7f3fb812af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.411+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2718145352 shutdown_connections 2024-03-29T11:20:51.391 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.411+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2718145352 >> 172.21.15.67:0/2718145352 conn(0x7f3fb8066a00 msgr2=0x7f3fb813ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:51.391 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.411+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2718145352 shutdown_connections 2024-03-29T11:20:51.391 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.411+0000 7f3fbe2ff640 1 -- 172.21.15.67:0/2718145352 wait complete. 2024-03-29T11:20:51.391 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:51.411+0000 7f3fbe2ff640 1 librados: shutdown 2024-03-29T11:20:51.403 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:58.210 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:58.362 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.384+0000 7fc94fb32640 10 monclient: get_monmap_and_config 2024-03-29T11:20:58.363 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.384+0000 7fc94fb32640 10 monclient: build_initial_monmap 2024-03-29T11:20:58.363 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.384+0000 7fc94fb32640 10 monclient: monmap: 2024-03-29T11:20:58.363 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:58.363 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:58.363 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:20:58.385686+0000 2024-03-29T11:20:58.363 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:20:58.385686+0000 2024-03-29T11:20:58.363 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:58.363 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:58.363 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:58.363 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:20:58.363 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:20:58.363 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:58.363 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.385+0000 7fc94fb32640 1 Processor -- start 2024-03-29T11:20:58.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.386+0000 7fc94fb32640 1 -- start start 2024-03-29T11:20:58.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.386+0000 7fc94fb32640 10 monclient: init 2024-03-29T11:20:58.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.386+0000 7fc94fb32640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:58.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.386+0000 7fc94fb32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:58.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.386+0000 7fc94fb32640 10 monclient(hunting): picked mon.noname-c con 0x7fc9480a9750 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:58.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.387+0000 7fc94fb32640 10 monclient(hunting): picked mon.noname-b con 0x7fc948149ac0 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:58.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.387+0000 7fc94fb32640 10 monclient(hunting): picked mon.noname-a con 0x7fc94814fe50 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:58.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.387+0000 7fc94fb32640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9480aa4a0 con 0x7fc94814fe50 2024-03-29T11:20:58.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.387+0000 7fc94fb32640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94814a680 con 0x7fc9480a9750 2024-03-29T11:20:58.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.387+0000 7fc94fb32640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9480af290 con 0x7fc948149ac0 2024-03-29T11:20:58.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.387+0000 7fc94fb32640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:58.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.387+0000 7fc94fb32640 10 monclient(hunting): authenticate will time out at 918.682948s 2024-03-29T11:20:58.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.390+0000 7fc94d8ce640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fc9480a9750 0x7fc9481436f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:48863/0 (socket says 172.21.15.67:48863) 2024-03-29T11:20:58.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.390+0000 7fc94d8ce640 1 -- 172.21.15.67:0/1678627162 learned_addr learned my addr 172.21.15.67:0/1678627162 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:58.369 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 1 -- 172.21.15.67:0/1678627162 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51811601 0 0) 0x7fc9480af290 con 0x7fc948149ac0 2024-03-29T11:20:58.369 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:58.369 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:58.369 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:58.369 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:20:58.370 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 1 -- 172.21.15.67:0/1678627162 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc934003520 con 0x7fc948149ac0 2024-03-29T11:20:58.372 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 1 -- 172.21.15.67:0/1678627162 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065234057 0 0) 0x7fc94814a680 con 0x7fc9480a9750 2024-03-29T11:20:58.372 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:58.372 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:58.372 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:58.372 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:20:58.372 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 1 -- 172.21.15.67:0/1678627162 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9480af290 con 0x7fc9480a9750 2024-03-29T11:20:58.372 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 1 -- 172.21.15.67:0/1678627162 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952703860 0 0) 0x7fc9480aa4a0 con 0x7fc94814fe50 2024-03-29T11:20:58.372 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:58.372 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:58.372 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:58.372 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:20:58.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.393+0000 7fc94c8cc640 1 -- 172.21.15.67:0/1678627162 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc94814a680 con 0x7fc94814fe50 2024-03-29T11:20:58.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94d0cd640 1 -- 172.21.15.67:0/1678627162 >> v1:172.21.15.67:6790/0 conn(0x7fc948149ac0 legacy=0x7fc948149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T11:20:58.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94d0cd640 1 -- 172.21.15.67:0/1678627162 >> v1:172.21.15.67:6790/0 conn(0x7fc948149ac0 legacy=0x7fc948149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:20:58.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94d0cd640 1 --1- 172.21.15.67:0/1678627162 >> v1:172.21.15.67:6790/0 conn(0x7fc948149ac0 0x7fc948149f70 :-1 s=OPENED pgs=161425 cs=1 l=1).handle_message read tag failed 2024-03-29T11:20:58.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94d0cd640 1 --1- 172.21.15.67:0/1678627162 >> v1:172.21.15.67:6790/0 conn(0x7fc948149ac0 0x7fc948149f70 :-1 s=OPENED pgs=161425 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:20:58.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.67:6790/0 2024-03-29T11:20:58.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 1 -- 172.21.15.67:0/1678627162 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 103966927 0 0) 0x7fc94814a680 con 0x7fc94814fe50 2024-03-29T11:20:58.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 1 -- 172.21.15.67:0/1678627162 >> v1:172.21.15.67:6790/0 conn(0x7fc948149ac0 legacy=0x7fc948149f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:20:58.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 1 -- 172.21.15.67:0/1678627162 >> v1:172.21.15.82:6789/0 conn(0x7fc9480a9750 legacy=0x7fc9481436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:58.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:58.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 1 monclient: found mon.noname-a 2024-03-29T11:20:58.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:58.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:20:58.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 1 -- 172.21.15.67:0/1678627162 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9480afb40 con 0x7fc94814fe50 2024-03-29T11:20:58.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 10 monclient: _finish_auth 0 2024-03-29T11:20:58.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:58.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:58.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94fb32640 5 monclient: authenticate success, global_id 4287 2024-03-29T11:20:58.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94fb32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:58.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 1 -- 172.21.15.67:0/1678627162 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc9440014e0 con 0x7fc94814fe50 2024-03-29T11:20:58.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:58.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.394+0000 7fc94c8cc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:58.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.395+0000 7fc94c8cc640 10 monclient: dump: 2024-03-29T11:20:58.374 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:58.374 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:58.374 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:58.374 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:58.375 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:58.375 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:58.375 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:58.375 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:58.375 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:58.375 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:58.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.395+0000 7fc94c8cc640 1 -- 172.21.15.67:0/1678627162 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc944001640 con 0x7fc94814fe50 2024-03-29T11:20:58.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.395+0000 7fc94c8cc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:58.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.395+0000 7fc94c8cc640 1 -- 172.21.15.67:0/1678627162 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc944001a90 con 0x7fc94814fe50 2024-03-29T11:20:58.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.395+0000 7fc94c8cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:58.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.395+0000 7fc94c8cc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:58.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.395+0000 7fc94c8cc640 10 monclient: dump: 2024-03-29T11:20:58.375 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:58.375 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:58.376 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:58.376 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:58.376 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:58.376 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:58.376 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:58.376 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:58.376 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:58.376 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:58.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.395+0000 7fc94fb32640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:58.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.395+0000 7fc94fb32640 4 set_mon_vals no callback set 2024-03-29T11:20:58.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.396+0000 7fc94fb32640 10 monclient: shutdown 2024-03-29T11:20:58.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.396+0000 7fc94fb32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:58.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.396+0000 7fc94fb32640 1 -- 172.21.15.67:0/1678627162 >> v1:172.21.15.67:6789/0 conn(0x7fc94814fe50 legacy=0x7fc948152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:58.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.396+0000 7fc94fb32640 1 -- 172.21.15.67:0/1678627162 shutdown_connections 2024-03-29T11:20:58.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.396+0000 7fc94fb32640 1 -- 172.21.15.67:0/1678627162 >> 172.21.15.67:0/1678627162 conn(0x7fc9480b35e0 msgr2=0x7fc9480b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:58.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.396+0000 7fc94fb32640 1 -- 172.21.15.67:0/1678627162 shutdown_connections 2024-03-29T11:20:58.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.396+0000 7fc94fb32640 1 -- 172.21.15.67:0/1678627162 wait complete. 2024-03-29T11:20:58.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.397+0000 7fc94fb32640 10 monclient: build_initial_monmap 2024-03-29T11:20:58.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.397+0000 7fc94fb32640 10 monclient: monmap: 2024-03-29T11:20:58.381 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:20:58.381 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:58.381 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:20:58.381 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:20:58.381 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:58.382 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:58.382 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:20:58.382 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:20:58.382 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:20:58.382 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:58.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.398+0000 7fc94fb32640 1 librados: starting msgr at 2024-03-29T11:20:58.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.398+0000 7fc94fb32640 1 librados: starting objecter 2024-03-29T11:20:58.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.398+0000 7fc94fb32640 1 Processor -- start 2024-03-29T11:20:58.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.398+0000 7fc94fb32640 1 -- start start 2024-03-29T11:20:58.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.398+0000 7fc94fb32640 1 librados: setting wanted keys 2024-03-29T11:20:58.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.398+0000 7fc94fb32640 1 librados: calling monclient init 2024-03-29T11:20:58.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.398+0000 7fc94fb32640 10 monclient: init 2024-03-29T11:20:58.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.399+0000 7fc94fb32640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:58.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.399+0000 7fc94fb32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:58.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.399+0000 7fc94fb32640 10 monclient(hunting): picked mon.noname-c con 0x7fc9480a9750 addr v1:172.21.15.67:6790/0 2024-03-29T11:20:58.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.399+0000 7fc94fb32640 10 monclient(hunting): picked mon.noname-a con 0x7fc948149ac0 addr v1:172.21.15.67:6789/0 2024-03-29T11:20:58.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.399+0000 7fc94fb32640 10 monclient(hunting): picked mon.noname-b con 0x7fc94814fe50 addr v1:172.21.15.82:6789/0 2024-03-29T11:20:58.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.399+0000 7fc94fb32640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9481db880 con 0x7fc948149ac0 2024-03-29T11:20:58.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.399+0000 7fc94fb32640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9481dc420 con 0x7fc94814fe50 2024-03-29T11:20:58.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.399+0000 7fc94fb32640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94813e790 con 0x7fc9480a9750 2024-03-29T11:20:58.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.399+0000 7fc94fb32640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:58.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.399+0000 7fc94fb32640 10 monclient(hunting): authenticate will time out at 918.695529s 2024-03-29T11:20:58.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.399+0000 7fc94d8ce640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fc9480a9750 0x7fc9481dbd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:42716/0 (socket says 172.21.15.67:42716) 2024-03-29T11:20:58.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.399+0000 7fc94d8ce640 1 -- 172.21.15.67:0/1093955661 learned_addr learned my addr 172.21.15.67:0/1093955661 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:20:58.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.399+0000 7fc93e7fc640 10 client.?.objecter ms_handle_connect 0x7fc9480a9750 2024-03-29T11:20:58.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.399+0000 7fc93e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:58.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.400+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033373180 0 0) 0x7fc94813e790 con 0x7fc9480a9750 2024-03-29T11:20:58.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.400+0000 7fc93e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:58.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.400+0000 7fc93e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:58.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.400+0000 7fc93e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:58.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.400+0000 7fc93e7fc640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:20:58.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.400+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc92c004eb0 con 0x7fc9480a9750 2024-03-29T11:20:58.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.400+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2335765109 0 0) 0x7fc92c004eb0 con 0x7fc9480a9750 2024-03-29T11:20:58.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.400+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc92c005300 con 0x7fc9480a9750 2024-03-29T11:20:58.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.400+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc938001870 con 0x7fc9480a9750 2024-03-29T11:20:58.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.400+0000 7fc93e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:58.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.400+0000 7fc93e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:58.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.400+0000 7fc93e7fc640 10 monclient(hunting): dump: 2024-03-29T11:20:58.385 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:58.385 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:58.385 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:58.385 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:58.385 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:58.385 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:58.385 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:58.385 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:58.385 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:58.385 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:58.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.400+0000 7fc93e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:58.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.401+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1383497213 0 0) 0x7fc92c005300 con 0x7fc9480a9750 2024-03-29T11:20:58.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.401+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 >> v1:172.21.15.82:6789/0 conn(0x7fc94814fe50 legacy=0x7fc9481e6090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:58.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.406+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 >> v1:172.21.15.67:6789/0 conn(0x7fc948149ac0 legacy=0x7fc94813f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:58.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.406+0000 7fc93e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:58.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.406+0000 7fc93e7fc640 1 monclient: found mon.c 2024-03-29T11:20:58.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.406+0000 7fc93e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:58.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.406+0000 7fc93e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:58.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.406+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94813e920 con 0x7fc9480a9750 2024-03-29T11:20:58.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.406+0000 7fc93e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:58.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.406+0000 7fc93e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:58.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.407+0000 7fc93e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:58.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.407+0000 7fc94fb32640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:20:58.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.407+0000 7fc94fb32640 10 monclient: _renew_subs 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.407+0000 7fc94fb32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.407+0000 7fc94fb32640 1 -- 172.21.15.67:0/1093955661 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94813b120 con 0x7fc9480a9750 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.407+0000 7fc94fb32640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.407+0000 7fc94fb32640 10 monclient: _renew_subs 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.407+0000 7fc94fb32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.407+0000 7fc94fb32640 1 -- 172.21.15.67:0/1093955661 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94813fc70 con 0x7fc9480a9750 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.407+0000 7fc94fb32640 1 librados: init done 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.408+0000 7fc94fb32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.408+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9380047e0 con 0x7fc9480a9750 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.408+0000 7fc94fb32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.408+0000 7fc94fb32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.408+0000 7fc94fb32640 1 -- 172.21.15.67:0/1093955661 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc948075f10 con 0x7fc9480a9750 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.409+0000 7fc93e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.409+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc938001870 con 0x7fc9480a9750 2024-03-29T11:20:58.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.409+0000 7fc93e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:58.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.409+0000 7fc93e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:58.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.409+0000 7fc93e7fc640 10 monclient: dump: 2024-03-29T11:20:58.390 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:20:58.390 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:20:58.390 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:58.390 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:20:58.390 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:58.390 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:20:58.390 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:20:58.390 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:20:58.390 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:20:58.390 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:20:58.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.409+0000 7fc93ffff640 4 set_mon_vals no callback set 2024-03-29T11:20:58.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.414+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fc93801a880 con 0x7fc9480a9750 2024-03-29T11:20:58.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.414+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fc938084790 con 0x7fc9480a9750 2024-03-29T11:20:58.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.414+0000 7fc93e7fc640 10 client.4283.objecter ms_dispatch 0x7fc948000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:58.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.414+0000 7fc93e7fc640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:58.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.414+0000 7fc93e7fc640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:58.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.415+0000 7fc93e7fc640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-29T11:20:58.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.415+0000 7fc94d0cd640 10 monclient: get_auth_request con 0x7fc92c06dff0 auth_method 0 2024-03-29T11:20:58.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.415+0000 7fc93e7fc640 10 client.4283.objecter ms_handle_connect 0x7fc92c06dff0 2024-03-29T11:20:58.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.417+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc93804fea0 con 0x7fc9480a9750 2024-03-29T11:20:58.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.417+0000 7fc93e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:58.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.417+0000 7fc93e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:58.731 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.754+0000 7fc94fb32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:58.731 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.754+0000 7fc94fb32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:58.732 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.754+0000 7fc94fb32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:20:58.732 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.754+0000 7fc94fb32640 1 -- 172.21.15.67:0/1093955661 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc94814a9d0 con 0x7fc9480a9750 2024-03-29T11:20:58.734 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.757+0000 7fc93e7fc640 1 -- 172.21.15.67:0/1093955661 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fc938051fa0 con 0x7fc9480a9750 2024-03-29T11:20:58.734 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.757+0000 7fc93e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:58.734 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.757+0000 7fc93e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:58.734 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:20:58.734 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:58.738 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.761+0000 7fc94fb32640 10 librados: watch_flush enter 2024-03-29T11:20:58.738 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.762+0000 7fc94fb32640 10 librados: watch_flush exit 2024-03-29T11:20:58.738 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.762+0000 7fc94fb32640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-29T11:20:58.738 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.762+0000 7fc94fb32640 10 client.4283.objecter successfully canceled tick 2024-03-29T11:20:58.738 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.762+0000 7fc94fb32640 1 -- 172.21.15.67:0/1093955661 >> v1:172.21.15.67:6816/404787197 conn(0x7fc92c06dff0 legacy=0x7fc92c0704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:58.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.762+0000 7fc94fb32640 10 monclient: shutdown 2024-03-29T11:20:58.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.762+0000 7fc94fb32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:58.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.762+0000 7fc94fb32640 1 -- 172.21.15.67:0/1093955661 >> v1:172.21.15.67:6790/0 conn(0x7fc9480a9750 legacy=0x7fc9481dbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:58.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.762+0000 7fc94fb32640 1 -- 172.21.15.67:0/1093955661 shutdown_connections 2024-03-29T11:20:58.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.762+0000 7fc94fb32640 1 -- 172.21.15.67:0/1093955661 >> 172.21.15.67:0/1093955661 conn(0x7fc9480b35e0 msgr2=0x7fc9480b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:58.740 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.762+0000 7fc94fb32640 1 -- 172.21.15.67:0/1093955661 shutdown_connections 2024-03-29T11:20:58.740 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.762+0000 7fc94fb32640 1 -- 172.21.15.67:0/1093955661 wait complete. 2024-03-29T11:20:58.740 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:20:58.763+0000 7fc94fb32640 1 librados: shutdown 2024-03-29T11:20:58.755 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:05.563 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:05.713 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.737+0000 7fb78e084640 10 monclient: get_monmap_and_config 2024-03-29T11:21:05.713 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.737+0000 7fb78e084640 10 monclient: build_initial_monmap 2024-03-29T11:21:05.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.737+0000 7fb78e084640 10 monclient: monmap: 2024-03-29T11:21:05.714 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:05.714 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:05.714 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:21:05.738050+0000 2024-03-29T11:21:05.714 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:21:05.738050+0000 2024-03-29T11:21:05.714 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:05.714 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:05.714 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:05.714 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:21:05.714 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:21:05.714 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:05.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.738+0000 7fb78e084640 1 Processor -- start 2024-03-29T11:21:05.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.738+0000 7fb78e084640 1 -- start start 2024-03-29T11:21:05.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.738+0000 7fb78e084640 10 monclient: init 2024-03-29T11:21:05.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.739+0000 7fb78e084640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:05.717 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.739+0000 7fb78e084640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:05.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.739+0000 7fb78e084640 10 monclient(hunting): picked mon.noname-b con 0x7fb7881457f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:05.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.742+0000 7fb78e084640 10 monclient(hunting): picked mon.noname-a con 0x7fb78813f650 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:05.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.742+0000 7fb78e084640 10 monclient(hunting): picked mon.noname-c con 0x7fb7880a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:05.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.742+0000 7fb78e084640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7880aaef0 con 0x7fb78813f650 2024-03-29T11:21:05.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.742+0000 7fb78e084640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb788145bd0 con 0x7fb7880a9d70 2024-03-29T11:21:05.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.742+0000 7fb78e084640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb788145eb0 con 0x7fb7881457f0 2024-03-29T11:21:05.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.742+0000 7fb78e084640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:05.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.742+0000 7fb78e084640 10 monclient(hunting): authenticate will time out at 926.038391s 2024-03-29T11:21:05.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.742+0000 7fb787fff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fb7880a9d70 0x7fb78814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:34972/0 (socket says 172.21.15.67:34972) 2024-03-29T11:21:05.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.742+0000 7fb787fff640 1 -- 172.21.15.67:0/1420105107 learned_addr learned my addr 172.21.15.67:0/1420105107 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:05.722 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.745+0000 7fb7867fc640 1 -- 172.21.15.67:0/1420105107 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220964144 0 0) 0x7fb788145bd0 con 0x7fb7880a9d70 2024-03-29T11:21:05.722 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.745+0000 7fb7867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:05.722 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.745+0000 7fb7867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:05.722 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.745+0000 7fb7867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:05.722 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.745+0000 7fb7867fc640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:21:05.722 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.745+0000 7fb7867fc640 1 -- 172.21.15.67:0/1420105107 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb770003520 con 0x7fb7880a9d70 2024-03-29T11:21:05.722 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 1 -- 172.21.15.67:0/1420105107 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704501525 0 0) 0x7fb788145eb0 con 0x7fb7881457f0 2024-03-29T11:21:05.723 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:05.723 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:05.723 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:05.723 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:21:05.723 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 1 -- 172.21.15.67:0/1420105107 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb788145bd0 con 0x7fb7881457f0 2024-03-29T11:21:05.724 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 1 -- 172.21.15.67:0/1420105107 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2762697688 0 0) 0x7fb770003520 con 0x7fb7880a9d70 2024-03-29T11:21:05.724 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 1 -- 172.21.15.67:0/1420105107 >> v1:172.21.15.67:6790/0 conn(0x7fb7881457f0 legacy=0x7fb78813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.724 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 1 -- 172.21.15.67:0/1420105107 >> v1:172.21.15.67:6789/0 conn(0x7fb78813f650 legacy=0x7fb7880b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.724 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:05.725 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 1 monclient: found mon.noname-c 2024-03-29T11:21:05.725 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:05.725 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:21:05.725 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 1 -- 172.21.15.67:0/1420105107 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb788146340 con 0x7fb7880a9d70 2024-03-29T11:21:05.725 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:05.725 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:05.725 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:05.725 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb78e084640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:21:05.726 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb78e084640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:05.726 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 1 -- 172.21.15.67:0/1420105107 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb780001a10 con 0x7fb7880a9d70 2024-03-29T11:21:05.726 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:05.726 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:05.726 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb7867fc640 10 monclient: dump: 2024-03-29T11:21:05.726 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:05.726 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:05.726 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:05.726 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:05.726 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:05.726 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:05.726 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:05.726 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:05.727 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:05.727 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:05.727 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.746+0000 7fb78e084640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:05.727 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.747+0000 7fb7867fc640 1 -- 172.21.15.67:0/1420105107 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb780001c30 con 0x7fb7880a9d70 2024-03-29T11:21:05.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.747+0000 7fb7867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:05.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.747+0000 7fb78e084640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:05.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.747+0000 7fb78e084640 4 set_mon_vals no callback set 2024-03-29T11:21:05.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.748+0000 7fb78e084640 10 monclient: shutdown 2024-03-29T11:21:05.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.748+0000 7fb78e084640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:05.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.748+0000 7fb78e084640 1 -- 172.21.15.67:0/1420105107 >> v1:172.21.15.82:6789/0 conn(0x7fb7880a9d70 legacy=0x7fb78814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.750+0000 7fb78e084640 1 -- 172.21.15.67:0/1420105107 shutdown_connections 2024-03-29T11:21:05.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.750+0000 7fb78e084640 1 -- 172.21.15.67:0/1420105107 >> 172.21.15.67:0/1420105107 conn(0x7fb788066a00 msgr2=0x7fb788066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:05.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.750+0000 7fb78e084640 1 -- 172.21.15.67:0/1420105107 shutdown_connections 2024-03-29T11:21:05.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.751+0000 7fb78e084640 1 -- 172.21.15.67:0/1420105107 wait complete. 2024-03-29T11:21:05.730 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.752+0000 7fb78e084640 10 monclient: build_initial_monmap 2024-03-29T11:21:05.730 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.752+0000 7fb78e084640 10 monclient: monmap: 2024-03-29T11:21:05.730 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:05.730 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:05.730 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:21:05.730 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:21:05.730 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.752+0000 7fb78e084640 1 librados: starting msgr at 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.752+0000 7fb78e084640 1 librados: starting objecter 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.752+0000 7fb78e084640 1 Processor -- start 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.752+0000 7fb78e084640 1 -- start start 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.752+0000 7fb78e084640 1 librados: setting wanted keys 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.752+0000 7fb78e084640 1 librados: calling monclient init 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.752+0000 7fb78e084640 10 monclient: init 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.753+0000 7fb78e084640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.753+0000 7fb78e084640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:05.731 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.753+0000 7fb78e084640 10 monclient(hunting): picked mon.noname-a con 0x7fb7880a9d70 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:05.732 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.753+0000 7fb78e084640 10 monclient(hunting): picked mon.noname-c con 0x7fb78813f650 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:05.732 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.753+0000 7fb78e084640 10 monclient(hunting): picked mon.noname-b con 0x7fb7881457f0 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:05.732 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.753+0000 7fb78e084640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb788142e30 con 0x7fb7880a9d70 2024-03-29T11:21:05.732 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.753+0000 7fb78e084640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7880abd10 con 0x7fb7881457f0 2024-03-29T11:21:05.732 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.754+0000 7fb78e084640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7880ad3e0 con 0x7fb78813f650 2024-03-29T11:21:05.732 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.754+0000 7fb78e084640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:05.732 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.754+0000 7fb78e084640 10 monclient(hunting): authenticate will time out at 926.049934s 2024-03-29T11:21:05.733 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.755+0000 7fb787fff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fb7881457f0 0x7fb7881e1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:36148/0 (socket says 172.21.15.67:36148) 2024-03-29T11:21:05.733 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.755+0000 7fb787fff640 1 -- 172.21.15.67:0/2989115143 learned_addr learned my addr 172.21.15.67:0/2989115143 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:05.733 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.756+0000 7fb76ffff640 10 client.?.objecter ms_handle_connect 0x7fb7881457f0 2024-03-29T11:21:05.733 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.756+0000 7fb76ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:05.733 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.756+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518244234 0 0) 0x7fb7880abd10 con 0x7fb7881457f0 2024-03-29T11:21:05.733 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.756+0000 7fb76ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:05.733 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.756+0000 7fb76ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:05.733 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.756+0000 7fb76ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:05.733 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.756+0000 7fb76ffff640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:21:05.734 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.756+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb774004eb0 con 0x7fb7881457f0 2024-03-29T11:21:05.734 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.757+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3903230723 0 0) 0x7fb774004eb0 con 0x7fb7881457f0 2024-03-29T11:21:05.734 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.757+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7880abd10 con 0x7fb7881457f0 2024-03-29T11:21:05.734 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.757+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb780004d00 con 0x7fb7881457f0 2024-03-29T11:21:05.734 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.757+0000 7fb76ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:05.734 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.757+0000 7fb76ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:05.734 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.757+0000 7fb76ffff640 10 monclient(hunting): dump: 2024-03-29T11:21:05.734 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:05.734 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:05.734 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:05.734 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:05.734 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:05.734 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:05.735 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:05.735 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:05.735 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:05.735 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:05.735 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.757+0000 7fb76ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:05.736 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.757+0000 7fb76ffff640 10 client.?.objecter ms_handle_connect 0x7fb78813f650 2024-03-29T11:21:05.736 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.757+0000 7fb76ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:05.737 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726578079 0 0) 0x7fb7880ad3e0 con 0x7fb78813f650 2024-03-29T11:21:05.737 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:05.737 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:05.737 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:05.737 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:21:05.737 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb774004eb0 con 0x7fb78813f650 2024-03-29T11:21:05.737 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2440392572 0 0) 0x7fb774004eb0 con 0x7fb78813f650 2024-03-29T11:21:05.737 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7740059e0 con 0x7fb78813f650 2024-03-29T11:21:05.737 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb77c0017f0 con 0x7fb78813f650 2024-03-29T11:21:05.737 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:05.737 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:05.737 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 10 monclient(hunting): dump: 2024-03-29T11:21:05.737 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:05.738 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:05.738 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:05.738 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:05.738 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:05.738 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:05.738 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:05.738 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:05.738 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:05.738 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:05.738 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 189205870 0 0) 0x7fb7740059e0 con 0x7fb78813f650 2024-03-29T11:21:05.738 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 >> v1:172.21.15.82:6789/0 conn(0x7fb7881457f0 legacy=0x7fb7881e1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.738 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 >> v1:172.21.15.67:6789/0 conn(0x7fb7880a9d70 legacy=0x7fb7881da9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:05.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 1 monclient: found mon.c 2024-03-29T11:21:05.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:05.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:05.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.758+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb788142a50 con 0x7fb78813f650 2024-03-29T11:21:05.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.759+0000 7fb76ffff640 10 monclient: _finish_auth 0 2024-03-29T11:21:05.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.759+0000 7fb76ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:05.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.759+0000 7fb76ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:05.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.759+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb77c001ce0 con 0x7fb78813f650 2024-03-29T11:21:05.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.759+0000 7fb76ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:05.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.759+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb77c0017f0 con 0x7fb78813f650 2024-03-29T11:21:05.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.759+0000 7fb76ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:05.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.759+0000 7fb76ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:05.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.759+0000 7fb76ffff640 10 monclient: dump: 2024-03-29T11:21:05.740 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:05.740 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:05.740 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:05.740 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:05.740 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:05.740 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:05.740 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:05.740 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:05.740 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:05.740 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:05.740 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.759+0000 7fb785ffb640 4 set_mon_vals no callback set 2024-03-29T11:21:05.741 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.760+0000 7fb78e084640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:21:05.741 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.760+0000 7fb78e084640 10 monclient: _renew_subs 2024-03-29T11:21:05.741 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.760+0000 7fb78e084640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:05.741 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.760+0000 7fb78e084640 1 -- 172.21.15.67:0/2989115143 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7880b0c40 con 0x7fb78813f650 2024-03-29T11:21:05.742 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.760+0000 7fb78e084640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:05.742 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.760+0000 7fb78e084640 10 monclient: _renew_subs 2024-03-29T11:21:05.742 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.760+0000 7fb78e084640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:05.742 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.760+0000 7fb78e084640 1 -- 172.21.15.67:0/2989115143 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7880b0fe0 con 0x7fb78813f650 2024-03-29T11:21:05.742 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.760+0000 7fb78e084640 1 librados: init done 2024-03-29T11:21:05.742 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.762+0000 7fb78e084640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:05.742 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.762+0000 7fb78e084640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:05.742 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.762+0000 7fb78e084640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:05.742 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.762+0000 7fb78e084640 1 -- 172.21.15.67:0/2989115143 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb788142ca0 con 0x7fb78813f650 2024-03-29T11:21:05.742 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.765+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fb77c0194e0 con 0x7fb78813f650 2024-03-29T11:21:05.743 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.766+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fb77c0832b0 con 0x7fb78813f650 2024-03-29T11:21:05.743 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.766+0000 7fb76ffff640 10 client.4289.objecter ms_dispatch 0x7fb788000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:05.743 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.766+0000 7fb76ffff640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:05.743 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.766+0000 7fb76ffff640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:05.743 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.766+0000 7fb76ffff640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-29T11:21:05.746 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.770+0000 7fb7877fe640 10 monclient: get_auth_request con 0x7fb77406e410 auth_method 0 2024-03-29T11:21:05.746 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.770+0000 7fb76ffff640 10 client.4289.objecter ms_handle_connect 0x7fb77406e410 2024-03-29T11:21:05.749 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.773+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb77c04e9c0 con 0x7fb78813f650 2024-03-29T11:21:05.749 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.773+0000 7fb76ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:05.749 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:05.773+0000 7fb76ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:06.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.107+0000 7fb78e084640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:06.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.107+0000 7fb78e084640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:06.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.107+0000 7fb78e084640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:06.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.107+0000 7fb78e084640 1 -- 172.21.15.67:0/2989115143 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7881db240 con 0x7fb78813f650 2024-03-29T11:21:06.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.108+0000 7fb76ffff640 1 -- 172.21.15.67:0/2989115143 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fb7881db240 con 0x7fb78813f650 2024-03-29T11:21:06.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.108+0000 7fb76ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:06.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.108+0000 7fb76ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:06.085 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:21:06.085 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:06.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.112+0000 7fb78e084640 10 librados: watch_flush enter 2024-03-29T11:21:06.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.112+0000 7fb78e084640 10 librados: watch_flush exit 2024-03-29T11:21:06.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.112+0000 7fb78e084640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-29T11:21:06.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.112+0000 7fb78e084640 10 client.4289.objecter successfully canceled tick 2024-03-29T11:21:06.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.112+0000 7fb78e084640 1 -- 172.21.15.67:0/2989115143 >> v1:172.21.15.67:6816/404787197 conn(0x7fb77406e410 legacy=0x7fb7740708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:06.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.112+0000 7fb78e084640 10 monclient: shutdown 2024-03-29T11:21:06.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.112+0000 7fb78e084640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:06.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.112+0000 7fb78e084640 1 -- 172.21.15.67:0/2989115143 >> v1:172.21.15.67:6790/0 conn(0x7fb78813f650 legacy=0x7fb7880ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:06.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.113+0000 7fb78e084640 1 -- 172.21.15.67:0/2989115143 shutdown_connections 2024-03-29T11:21:06.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.114+0000 7fb78e084640 1 -- 172.21.15.67:0/2989115143 >> 172.21.15.67:0/2989115143 conn(0x7fb788066a00 msgr2=0x7fb78813cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:06.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.114+0000 7fb78e084640 1 -- 172.21.15.67:0/2989115143 shutdown_connections 2024-03-29T11:21:06.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.114+0000 7fb78e084640 1 -- 172.21.15.67:0/2989115143 wait complete. 2024-03-29T11:21:06.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:06.114+0000 7fb78e084640 1 librados: shutdown 2024-03-29T11:21:06.109 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:12.915 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:13.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.087+0000 7f070906a640 10 monclient: get_monmap_and_config 2024-03-29T11:21:13.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.087+0000 7f070906a640 10 monclient: build_initial_monmap 2024-03-29T11:21:13.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.087+0000 7f070906a640 10 monclient: monmap: 2024-03-29T11:21:13.069 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:13.069 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:13.069 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:21:13.088389+0000 2024-03-29T11:21:13.069 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:21:13.088389+0000 2024-03-29T11:21:13.069 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:13.069 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:13.069 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:13.069 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:21:13.069 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:21:13.069 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:13.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.088+0000 7f070906a640 1 Processor -- start 2024-03-29T11:21:13.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.088+0000 7f070906a640 1 -- start start 2024-03-29T11:21:13.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.088+0000 7f070906a640 10 monclient: init 2024-03-29T11:21:13.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.089+0000 7f070906a640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:13.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.089+0000 7f070906a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:13.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.089+0000 7f070906a640 10 monclient(hunting): picked mon.noname-a con 0x7f07040ad6c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:13.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.089+0000 7f070906a640 10 monclient(hunting): picked mon.noname-b con 0x7f07040af6c0 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:13.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.089+0000 7f070906a640 10 monclient(hunting): picked mon.noname-c con 0x7f070414fdc0 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:13.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.089+0000 7f070906a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07040a94e0 con 0x7f07040ad6c0 2024-03-29T11:21:13.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.089+0000 7f070906a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07040af3a0 con 0x7f070414fdc0 2024-03-29T11:21:13.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.089+0000 7f070906a640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07040afb60 con 0x7f07040af6c0 2024-03-29T11:21:13.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.089+0000 7f070906a640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:13.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.089+0000 7f070906a640 10 monclient(hunting): authenticate will time out at 933.385291s 2024-03-29T11:21:13.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.092+0000 7f070259c640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f07040af6c0 0x7f0704152980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:55743/0 (socket says 172.21.15.67:55743) 2024-03-29T11:21:13.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.092+0000 7f070259c640 1 -- 172.21.15.67:0/4067909199 learned_addr learned my addr 172.21.15.67:0/4067909199 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:13.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.094+0000 7f0701d9b640 1 -- 172.21.15.67:0/4067909199 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125620836 0 0) 0x7f07040afb60 con 0x7f07040af6c0 2024-03-29T11:21:13.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.094+0000 7f0701d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:13.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.094+0000 7f0701d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:13.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.094+0000 7f0701d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:13.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.094+0000 7f0701d9b640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:21:13.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.094+0000 7f0701d9b640 1 -- 172.21.15.67:0/4067909199 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06ec003520 con 0x7f07040af6c0 2024-03-29T11:21:13.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.094+0000 7f0701d9b640 1 -- 172.21.15.67:0/4067909199 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4222303889 0 0) 0x7f06ec003520 con 0x7f07040af6c0 2024-03-29T11:21:13.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.094+0000 7f0701d9b640 1 -- 172.21.15.67:0/4067909199 >> v1:172.21.15.82:6789/0 conn(0x7f070414fdc0 legacy=0x7f070415a8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:13.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.095+0000 7f0701d9b640 1 -- 172.21.15.67:0/4067909199 >> v1:172.21.15.67:6789/0 conn(0x7f07040ad6c0 legacy=0x7f070414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.095+0000 7f0701d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:21:13.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.095+0000 7f0701d9b640 1 monclient: found mon.noname-b 2024-03-29T11:21:13.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.095+0000 7f0701d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:13.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.095+0000 7f0701d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:21:13.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.095+0000 7f0701d9b640 1 -- 172.21.15.67:0/4067909199 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07040aab80 con 0x7f07040af6c0 2024-03-29T11:21:13.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.095+0000 7f0701d9b640 10 monclient: _finish_auth 0 2024-03-29T11:21:13.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.095+0000 7f0701d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:21:13.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.095+0000 7f0701d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:13.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.095+0000 7f0701d9b640 1 -- 172.21.15.67:0/4067909199 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f06fc001670 con 0x7f07040af6c0 2024-03-29T11:21:13.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.095+0000 7f0701d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:13.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.095+0000 7f0701d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:13.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.095+0000 7f0701d9b640 10 monclient: dump: 2024-03-29T11:21:13.074 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:13.074 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:13.074 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:13.074 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:13.074 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:13.074 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:13.074 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:13.074 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:13.074 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:13.074 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:13.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.096+0000 7f070906a640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:21:13.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.096+0000 7f070906a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:13.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.097+0000 7f0701d9b640 1 -- 172.21.15.67:0/4067909199 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06fc0018d0 con 0x7f07040af6c0 2024-03-29T11:21:13.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.097+0000 7f0701d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:13.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.097+0000 7f070906a640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:13.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.097+0000 7f070906a640 4 set_mon_vals no callback set 2024-03-29T11:21:13.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.097+0000 7f070906a640 10 monclient: shutdown 2024-03-29T11:21:13.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.097+0000 7f070906a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:13.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.097+0000 7f070906a640 1 -- 172.21.15.67:0/4067909199 >> v1:172.21.15.67:6790/0 conn(0x7f07040af6c0 legacy=0x7f0704152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.098+0000 7f070906a640 1 -- 172.21.15.67:0/4067909199 shutdown_connections 2024-03-29T11:21:13.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.098+0000 7f070906a640 1 -- 172.21.15.67:0/4067909199 >> 172.21.15.67:0/4067909199 conn(0x7f07040b35e0 msgr2=0x7f07040b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:13.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.098+0000 7f070906a640 1 -- 172.21.15.67:0/4067909199 shutdown_connections 2024-03-29T11:21:13.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.098+0000 7f070906a640 1 -- 172.21.15.67:0/4067909199 wait complete. 2024-03-29T11:21:13.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.099+0000 7f070906a640 10 monclient: build_initial_monmap 2024-03-29T11:21:13.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.099+0000 7f070906a640 10 monclient: monmap: 2024-03-29T11:21:13.080 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:13.080 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:13.080 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:21:13.080 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:21:13.080 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:13.081 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:13.081 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:13.081 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:21:13.081 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:21:13.081 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:13.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.099+0000 7f070906a640 1 librados: starting msgr at 2024-03-29T11:21:13.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.099+0000 7f070906a640 1 librados: starting objecter 2024-03-29T11:21:13.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.100+0000 7f070906a640 1 Processor -- start 2024-03-29T11:21:13.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.100+0000 7f070906a640 1 -- start start 2024-03-29T11:21:13.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.100+0000 7f070906a640 1 librados: setting wanted keys 2024-03-29T11:21:13.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.100+0000 7f070906a640 1 librados: calling monclient init 2024-03-29T11:21:13.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.100+0000 7f070906a640 10 monclient: init 2024-03-29T11:21:13.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.100+0000 7f070906a640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:13.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.100+0000 7f070906a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:13.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.101+0000 7f070906a640 10 monclient(hunting): picked mon.noname-a con 0x7f07040ad6c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:13.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.101+0000 7f070906a640 10 monclient(hunting): picked mon.noname-c con 0x7f07040af6c0 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:13.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.101+0000 7f070906a640 10 monclient(hunting): picked mon.noname-b con 0x7f070414fdc0 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:13.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.101+0000 7f070906a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07041f8260 con 0x7f07040ad6c0 2024-03-29T11:21:13.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.101+0000 7f070906a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070407d850 con 0x7f070414fdc0 2024-03-29T11:21:13.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.101+0000 7f070906a640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070407c5a0 con 0x7f07040af6c0 2024-03-29T11:21:13.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.101+0000 7f070906a640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:13.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.101+0000 7f070906a640 10 monclient(hunting): authenticate will time out at 933.396985s 2024-03-29T11:21:13.086 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.104+0000 7f0702d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f07040ad6c0 0x7f070407be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44971/0 (socket says 172.21.15.67:44971) 2024-03-29T11:21:13.086 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.104+0000 7f0702d9d640 1 -- 172.21.15.67:0/1987908347 learned_addr learned my addr 172.21.15.67:0/1987908347 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:13.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 10 client.?.objecter ms_handle_connect 0x7f07040ad6c0 2024-03-29T11:21:13.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:13.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318492821 0 0) 0x7f07041f8260 con 0x7f07040ad6c0 2024-03-29T11:21:13.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:13.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:13.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:13.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:21:13.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f0004eb0 con 0x7f07040ad6c0 2024-03-29T11:21:13.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2508554338 0 0) 0x7f06f0004eb0 con 0x7f07040ad6c0 2024-03-29T11:21:13.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06f0005300 con 0x7f07040ad6c0 2024-03-29T11:21:13.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f06f4001970 con 0x7f07040ad6c0 2024-03-29T11:21:13.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:13.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:13.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:13.088 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:13.088 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:13.088 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:13.088 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:13.088 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:13.088 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:13.088 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:13.088 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:13.088 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:13.088 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:13.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:13.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3111765518 0 0) 0x7f06f0005300 con 0x7f07040ad6c0 2024-03-29T11:21:13.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.105+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 >> v1:172.21.15.67:6790/0 conn(0x7f07040af6c0 legacy=0x7f07041f8710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:13.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 >> v1:172.21.15.82:6789/0 conn(0x7f070414fdc0 legacy=0x7f07041ff800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f06eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:13.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f06eb7fe640 1 monclient: found mon.a 2024-03-29T11:21:13.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f06eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:13.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f06eb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:21:13.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0704070160 con 0x7f07040ad6c0 2024-03-29T11:21:13.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f06eb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:13.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f06eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:13.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f06eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:13.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f06eb7fe640 10 client.?.objecter ms_handle_connect 0x7f070414fdc0 2024-03-29T11:21:13.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f06eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:13.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f070906a640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:21:13.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f070906a640 10 monclient: _renew_subs 2024-03-29T11:21:13.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f070906a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:21:13.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f070906a640 1 -- 172.21.15.67:0/1987908347 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07041f8e20 con 0x7f07040ad6c0 2024-03-29T11:21:13.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f070906a640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:13.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f070906a640 10 monclient: _renew_subs 2024-03-29T11:21:13.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f070906a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:21:13.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f070906a640 1 -- 172.21.15.67:0/1987908347 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07041f9170 con 0x7f07040ad6c0 2024-03-29T11:21:13.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.108+0000 7f070906a640 1 librados: init done 2024-03-29T11:21:13.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.109+0000 7f070906a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:13.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.109+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06f4001c50 con 0x7f07040ad6c0 2024-03-29T11:21:13.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.109+0000 7f070906a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:13.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.109+0000 7f070906a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:21:13.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.109+0000 7f070906a640 1 -- 172.21.15.67:0/1987908347 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f070407c730 con 0x7f07040ad6c0 2024-03-29T11:21:13.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.109+0000 7f06eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:13.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.109+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f06f4001970 con 0x7f07040ad6c0 2024-03-29T11:21:13.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.109+0000 7f06eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:13.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.109+0000 7f06eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:13.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.109+0000 7f06eb7fe640 10 monclient: dump: 2024-03-29T11:21:13.091 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:13.091 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:13.091 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:13.091 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:13.091 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:13.091 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:13.091 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:13.091 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:13.091 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:13.091 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:13.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.110+0000 7f0700d99640 4 set_mon_vals no callback set 2024-03-29T11:21:13.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.111+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f06f4003940 con 0x7f07040ad6c0 2024-03-29T11:21:13.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.112+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f06f4084860 con 0x7f07040ad6c0 2024-03-29T11:21:13.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.112+0000 7f06eb7fe640 10 client.4296.objecter ms_dispatch 0x7f0704000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:13.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.112+0000 7f06eb7fe640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:13.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.112+0000 7f06eb7fe640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:13.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.112+0000 7f06eb7fe640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-29T11:21:13.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.113+0000 7f070259c640 10 monclient: get_auth_request con 0x7f06f006dfb0 auth_method 0 2024-03-29T11:21:13.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.114+0000 7f06eb7fe640 10 client.4296.objecter ms_handle_connect 0x7f06f006dfb0 2024-03-29T11:21:13.098 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.119+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f070407c730 con 0x7f07040ad6c0 2024-03-29T11:21:13.098 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.119+0000 7f06eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:13.098 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.119+0000 7f06eb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:13.418 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.441+0000 7f070906a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:13.418 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.441+0000 7f070906a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:13.418 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.441+0000 7f070906a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:21:13.418 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.441+0000 7f070906a640 1 -- 172.21.15.67:0/1987908347 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07041f8430 con 0x7f07040ad6c0 2024-03-29T11:21:13.419 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.442+0000 7f06eb7fe640 1 -- 172.21.15.67:0/1987908347 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f07041f8430 con 0x7f07040ad6c0 2024-03-29T11:21:13.419 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.442+0000 7f06eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:13.419 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.442+0000 7f06eb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:13.419 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:21:13.420 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:13.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.446+0000 7f070906a640 10 librados: watch_flush enter 2024-03-29T11:21:13.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.446+0000 7f070906a640 10 librados: watch_flush exit 2024-03-29T11:21:13.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.446+0000 7f070906a640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-29T11:21:13.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.446+0000 7f070906a640 10 client.4296.objecter successfully canceled tick 2024-03-29T11:21:13.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.446+0000 7f070906a640 1 -- 172.21.15.67:0/1987908347 >> v1:172.21.15.67:6816/404787197 conn(0x7f06f006dfb0 legacy=0x7f06f0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.446+0000 7f070906a640 10 monclient: shutdown 2024-03-29T11:21:13.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.446+0000 7f070906a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:13.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.446+0000 7f070906a640 1 -- 172.21.15.67:0/1987908347 >> v1:172.21.15.67:6789/0 conn(0x7f07040ad6c0 legacy=0x7f070407be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.447+0000 7f070906a640 1 -- 172.21.15.67:0/1987908347 shutdown_connections 2024-03-29T11:21:13.424 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.447+0000 7f070906a640 1 -- 172.21.15.67:0/1987908347 >> 172.21.15.67:0/1987908347 conn(0x7f07040b35e0 msgr2=0x7f07040b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:13.424 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.447+0000 7f070906a640 1 -- 172.21.15.67:0/1987908347 shutdown_connections 2024-03-29T11:21:13.424 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.448+0000 7f070906a640 1 -- 172.21.15.67:0/1987908347 wait complete. 2024-03-29T11:21:13.425 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:13.448+0000 7f070906a640 1 librados: shutdown 2024-03-29T11:21:13.442 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:20.250 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:20.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.435+0000 7f667c99f640 10 monclient: get_monmap_and_config 2024-03-29T11:21:20.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.435+0000 7f667c99f640 10 monclient: build_initial_monmap 2024-03-29T11:21:20.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.435+0000 7f667c99f640 10 monclient: monmap: 2024-03-29T11:21:20.412 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:20.412 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:20.412 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:21:20.436621+0000 2024-03-29T11:21:20.412 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:21:20.436621+0000 2024-03-29T11:21:20.412 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:20.412 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:20.413 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:20.413 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:21:20.413 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:21:20.413 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:20.414 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.438+0000 7f667c99f640 1 Processor -- start 2024-03-29T11:21:20.417 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.438+0000 7f667c99f640 1 -- start start 2024-03-29T11:21:20.417 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.438+0000 7f667c99f640 10 monclient: init 2024-03-29T11:21:20.417 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.439+0000 7f667c99f640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:20.417 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.439+0000 7f667c99f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:20.418 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.439+0000 7f667c99f640 10 monclient(hunting): picked mon.noname-b con 0x7f66780b2800 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:20.418 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.439+0000 7f667c99f640 10 monclient(hunting): picked mon.noname-a con 0x7f6678142b20 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:20.418 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.439+0000 7f667c99f640 10 monclient(hunting): picked mon.noname-c con 0x7f667813c040 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:20.418 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.439+0000 7f667c99f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780aad10 con 0x7f6678142b20 2024-03-29T11:21:20.418 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.439+0000 7f667c99f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6678141f30 con 0x7f667813c040 2024-03-29T11:21:20.418 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.439+0000 7f667c99f640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780aaf60 con 0x7f66780b2800 2024-03-29T11:21:20.418 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.439+0000 7f667c99f640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:20.418 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.439+0000 7f667c99f640 10 monclient(hunting): authenticate will time out at 940.735331s 2024-03-29T11:21:20.420 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.444+0000 7f66777fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f66780b2800 0x7f6678142410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:49209/0 (socket says 172.21.15.67:49209) 2024-03-29T11:21:20.420 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.444+0000 7f66777fe640 1 -- 172.21.15.67:0/2630196827 learned_addr learned my addr 172.21.15.67:0/2630196827 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:20.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.446+0000 7f66767fc640 1 -- 172.21.15.67:0/2630196827 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193820409 0 0) 0x7f66780aaf60 con 0x7f66780b2800 2024-03-29T11:21:20.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.446+0000 7f66767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:20.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.446+0000 7f66767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:20.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.446+0000 7f66767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:20.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.446+0000 7f66767fc640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:21:20.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.446+0000 7f66767fc640 1 -- 172.21.15.67:0/2630196827 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6660003520 con 0x7f66780b2800 2024-03-29T11:21:20.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.446+0000 7f66767fc640 1 -- 172.21.15.67:0/2630196827 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016761449 0 0) 0x7f66780aad10 con 0x7f6678142b20 2024-03-29T11:21:20.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.446+0000 7f66767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:20.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.446+0000 7f66767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:20.423 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.446+0000 7f66767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:20.424 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.446+0000 7f66767fc640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:21:20.424 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.446+0000 7f66767fc640 1 -- 172.21.15.67:0/2630196827 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66780aaf60 con 0x7f6678142b20 2024-03-29T11:21:20.426 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 1 -- 172.21.15.67:0/2630196827 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1132832991 0 0) 0x7f6660003520 con 0x7f66780b2800 2024-03-29T11:21:20.426 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 1 -- 172.21.15.67:0/2630196827 >> v1:172.21.15.82:6789/0 conn(0x7f667813c040 legacy=0x7f667814d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:20.426 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 1 -- 172.21.15.67:0/2630196827 >> v1:172.21.15.67:6789/0 conn(0x7f6678142b20 legacy=0x7f667813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:20.426 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:20.426 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 1 monclient: found mon.noname-b 2024-03-29T11:21:20.426 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:20.426 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:21:20.426 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 1 -- 172.21.15.67:0/2630196827 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780b3900 con 0x7f66780b2800 2024-03-29T11:21:20.426 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:20.426 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:20.426 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:20.427 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f667c99f640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:21:20.427 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f667c99f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:20.427 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 1 -- 172.21.15.67:0/2630196827 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f666c001690 con 0x7f66780b2800 2024-03-29T11:21:20.427 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:20.427 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:20.427 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f66767fc640 10 monclient: dump: 2024-03-29T11:21:20.427 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:20.427 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:20.427 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:20.427 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:20.427 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:20.427 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:20.427 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:20.427 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:20.428 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:20.428 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:20.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.449+0000 7f667c99f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:20.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.450+0000 7f66767fc640 1 -- 172.21.15.67:0/2630196827 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f666c001870 con 0x7f66780b2800 2024-03-29T11:21:20.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.450+0000 7f66767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:20.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.450+0000 7f66767fc640 1 -- 172.21.15.67:0/2630196827 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f666c001c80 con 0x7f66780b2800 2024-03-29T11:21:20.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.450+0000 7f66767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:20.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.450+0000 7f66767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:20.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.450+0000 7f66767fc640 10 monclient: dump: 2024-03-29T11:21:20.429 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:20.429 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:20.429 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:20.429 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:20.429 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:20.429 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:20.429 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:20.429 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:20.429 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:20.429 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:20.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.450+0000 7f667c99f640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:20.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.450+0000 7f667c99f640 4 set_mon_vals no callback set 2024-03-29T11:21:20.430 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.451+0000 7f667c99f640 10 monclient: shutdown 2024-03-29T11:21:20.430 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.451+0000 7f667c99f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:20.430 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.451+0000 7f667c99f640 1 -- 172.21.15.67:0/2630196827 >> v1:172.21.15.67:6790/0 conn(0x7f66780b2800 legacy=0x7f6678142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:20.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.453+0000 7f667c99f640 1 -- 172.21.15.67:0/2630196827 shutdown_connections 2024-03-29T11:21:20.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.453+0000 7f667c99f640 1 -- 172.21.15.67:0/2630196827 >> 172.21.15.67:0/2630196827 conn(0x7f667806d0e0 msgr2=0x7f6678066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:20.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.454+0000 7f667c99f640 1 -- 172.21.15.67:0/2630196827 shutdown_connections 2024-03-29T11:21:20.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.454+0000 7f667c99f640 1 -- 172.21.15.67:0/2630196827 wait complete. 2024-03-29T11:21:20.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.455+0000 7f667c99f640 10 monclient: build_initial_monmap 2024-03-29T11:21:20.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.455+0000 7f667c99f640 10 monclient: monmap: 2024-03-29T11:21:20.431 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:20.431 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:20.432 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:21:20.432 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:21:20.432 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:20.432 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:20.432 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:20.432 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:21:20.432 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:21:20.432 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:20.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.455+0000 7f667c99f640 1 librados: starting msgr at 2024-03-29T11:21:20.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.455+0000 7f667c99f640 1 librados: starting objecter 2024-03-29T11:21:20.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.456+0000 7f667c99f640 1 Processor -- start 2024-03-29T11:21:20.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.456+0000 7f667c99f640 1 -- start start 2024-03-29T11:21:20.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.456+0000 7f667c99f640 1 librados: setting wanted keys 2024-03-29T11:21:20.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.456+0000 7f667c99f640 1 librados: calling monclient init 2024-03-29T11:21:20.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.456+0000 7f667c99f640 10 monclient: init 2024-03-29T11:21:20.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.457+0000 7f667c99f640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:20.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.457+0000 7f667c99f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:20.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.457+0000 7f667c99f640 10 monclient(hunting): picked mon.noname-c con 0x7f667813c040 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:20.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.457+0000 7f667c99f640 10 monclient(hunting): picked mon.noname-a con 0x7f6678142b20 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:20.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.457+0000 7f667c99f640 10 monclient(hunting): picked mon.noname-b con 0x7f66781386f0 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:20.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.457+0000 7f667c99f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780ac3f0 con 0x7f6678142b20 2024-03-29T11:21:20.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.457+0000 7f667c99f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780abf30 con 0x7f66781386f0 2024-03-29T11:21:20.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.457+0000 7f667c99f640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780ad160 con 0x7f667813c040 2024-03-29T11:21:20.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.457+0000 7f667c99f640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:20.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.457+0000 7f667c99f640 10 monclient(hunting): authenticate will time out at 940.753170s 2024-03-29T11:21:20.441 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.461+0000 7f6677fff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f66781386f0 0x7f66781e29d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:52259/0 (socket says 172.21.15.67:52259) 2024-03-29T11:21:20.441 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.461+0000 7f6677fff640 1 -- 172.21.15.67:0/3939398132 learned_addr learned my addr 172.21.15.67:0/3939398132 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:20.441 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.461+0000 7f665ffff640 10 client.?.objecter ms_handle_connect 0x7f66781386f0 2024-03-29T11:21:20.441 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.461+0000 7f665ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:20.441 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.461+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835899669 0 0) 0x7f66780abf30 con 0x7f66781386f0 2024-03-29T11:21:20.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.461+0000 7f665ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:20.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.461+0000 7f665ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:20.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.461+0000 7f665ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:20.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.461+0000 7f665ffff640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:21:20.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.461+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6664004eb0 con 0x7f66781386f0 2024-03-29T11:21:20.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.462+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1180185042 0 0) 0x7f6664004eb0 con 0x7f66781386f0 2024-03-29T11:21:20.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.462+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66780abf30 con 0x7f66781386f0 2024-03-29T11:21:20.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.462+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6670001d00 con 0x7f66781386f0 2024-03-29T11:21:20.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.462+0000 7f665ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:20.442 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.462+0000 7f665ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:20.443 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.462+0000 7f665ffff640 10 monclient(hunting): dump: 2024-03-29T11:21:20.443 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:20.443 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:20.443 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:20.443 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:20.443 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:20.443 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:20.443 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:20.443 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:20.443 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:20.443 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:20.443 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.462+0000 7f665ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:20.443 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.462+0000 7f665ffff640 10 client.?.objecter ms_handle_connect 0x7f667813c040 2024-03-29T11:21:20.443 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.462+0000 7f665ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.462+0000 7f665ffff640 10 client.?.objecter ms_handle_connect 0x7f6678142b20 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.463+0000 7f665ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.463+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049990228 0 0) 0x7f66780ac3f0 con 0x7f6678142b20 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.463+0000 7f665ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.463+0000 7f665ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.463+0000 7f665ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.463+0000 7f665ffff640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.463+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6664004eb0 con 0x7f6678142b20 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.463+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3080792680 0 0) 0x7f66780abf30 con 0x7f66781386f0 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.463+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 >> v1:172.21.15.67:6790/0 conn(0x7f667813c040 legacy=0x7f6678137fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.463+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 >> v1:172.21.15.67:6789/0 conn(0x7f6678142b20 legacy=0x7f6678143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.463+0000 7f665ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.463+0000 7f665ffff640 1 monclient: found mon.b 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.463+0000 7f665ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:20.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.463+0000 7f665ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:21:20.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.464+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780ad440 con 0x7f66781386f0 2024-03-29T11:21:20.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.464+0000 7f665ffff640 10 monclient: _finish_auth 0 2024-03-29T11:21:20.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.464+0000 7f665ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:20.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.464+0000 7f665ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:20.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.464+0000 7f667c99f640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:21:20.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.464+0000 7f667c99f640 10 monclient: _renew_subs 2024-03-29T11:21:20.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.464+0000 7f667c99f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:21:20.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.464+0000 7f667c99f640 1 -- 172.21.15.67:0/3939398132 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66780b1040 con 0x7f66781386f0 2024-03-29T11:21:20.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.464+0000 7f667c99f640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:20.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.464+0000 7f667c99f640 10 monclient: _renew_subs 2024-03-29T11:21:20.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.464+0000 7f667c99f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:21:20.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.464+0000 7f667c99f640 1 -- 172.21.15.67:0/3939398132 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6678143da0 con 0x7f66781386f0 2024-03-29T11:21:20.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.464+0000 7f667c99f640 1 librados: init done 2024-03-29T11:21:20.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.465+0000 7f667c99f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:20.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.465+0000 7f667c99f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:20.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.465+0000 7f667c99f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:21:20.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.465+0000 7f667c99f640 1 -- 172.21.15.67:0/3939398132 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66780ac1e0 con 0x7f66781386f0 2024-03-29T11:21:20.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.466+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6670002080 con 0x7f66781386f0 2024-03-29T11:21:20.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.466+0000 7f665ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:20.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.466+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6670003de0 con 0x7f66781386f0 2024-03-29T11:21:20.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.466+0000 7f665ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:20.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.466+0000 7f66757fa640 4 set_mon_vals no callback set 2024-03-29T11:21:20.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.467+0000 7f665ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:20.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.467+0000 7f665ffff640 10 monclient: dump: 2024-03-29T11:21:20.447 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:20.447 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.467+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f6670003fc0 con 0x7f66781386f0 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.468+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f6670084eb0 con 0x7f66781386f0 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.468+0000 7f665ffff640 10 client.4282.objecter ms_dispatch 0x7f6678000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.468+0000 7f665ffff640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.468+0000 7f665ffff640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.468+0000 7f665ffff640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-29T11:21:20.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.469+0000 7f66777fe640 10 monclient: get_auth_request con 0x7f666406dde0 auth_method 0 2024-03-29T11:21:20.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.469+0000 7f665ffff640 10 client.4282.objecter ms_handle_connect 0x7f666406dde0 2024-03-29T11:21:20.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.472+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6670050790 con 0x7f66781386f0 2024-03-29T11:21:20.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.473+0000 7f665ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:20.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.473+0000 7f665ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:20.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.807+0000 7f667c99f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:20.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.807+0000 7f667c99f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:20.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.807+0000 7f667c99f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:21:20.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.807+0000 7f667c99f640 1 -- 172.21.15.67:0/3939398132 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f667804f730 con 0x7f66781386f0 2024-03-29T11:21:20.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.809+0000 7f665ffff640 1 -- 172.21.15.67:0/3939398132 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f66700526c0 con 0x7f66781386f0 2024-03-29T11:21:20.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.809+0000 7f665ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:20.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.809+0000 7f665ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:20.785 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:21:20.785 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:20.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.812+0000 7f667c99f640 10 librados: watch_flush enter 2024-03-29T11:21:20.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.812+0000 7f667c99f640 10 librados: watch_flush exit 2024-03-29T11:21:20.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.813+0000 7f667c99f640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-29T11:21:20.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.813+0000 7f667c99f640 10 client.4282.objecter successfully canceled tick 2024-03-29T11:21:20.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.813+0000 7f667c99f640 1 -- 172.21.15.67:0/3939398132 >> v1:172.21.15.67:6816/404787197 conn(0x7f666406dde0 legacy=0x7f66640702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:20.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.813+0000 7f667c99f640 10 monclient: shutdown 2024-03-29T11:21:20.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.813+0000 7f667c99f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:20.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.813+0000 7f667c99f640 1 -- 172.21.15.67:0/3939398132 >> v1:172.21.15.82:6789/0 conn(0x7f66781386f0 legacy=0x7f66781e29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:20.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.813+0000 7f667c99f640 1 -- 172.21.15.67:0/3939398132 shutdown_connections 2024-03-29T11:21:20.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.813+0000 7f667c99f640 1 -- 172.21.15.67:0/3939398132 >> 172.21.15.67:0/3939398132 conn(0x7f667806d0e0 msgr2=0x7f66780b6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:20.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.818+0000 7f667c99f640 1 -- 172.21.15.67:0/3939398132 shutdown_connections 2024-03-29T11:21:20.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.818+0000 7f667c99f640 1 -- 172.21.15.67:0/3939398132 wait complete. 2024-03-29T11:21:20.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:20.818+0000 7f667c99f640 1 librados: shutdown 2024-03-29T11:21:20.811 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:27.616 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:27.767 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.791+0000 7f0520474640 10 monclient: get_monmap_and_config 2024-03-29T11:21:27.767 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.791+0000 7f0520474640 10 monclient: build_initial_monmap 2024-03-29T11:21:27.767 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.791+0000 7f0520474640 10 monclient: monmap: 2024-03-29T11:21:27.767 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:27.768 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:27.768 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:21:27.792142+0000 2024-03-29T11:21:27.768 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:21:27.792142+0000 2024-03-29T11:21:27.768 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:27.768 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:27.768 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:27.768 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:21:27.768 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:21:27.768 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:27.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.793+0000 7f0520474640 1 Processor -- start 2024-03-29T11:21:27.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.793+0000 7f0520474640 1 -- start start 2024-03-29T11:21:27.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.793+0000 7f0520474640 10 monclient: init 2024-03-29T11:21:27.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.794+0000 7f0520474640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:27.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.794+0000 7f0520474640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:27.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.794+0000 7f0520474640 10 monclient(hunting): picked mon.noname-c con 0x7f05180b1f60 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:27.770 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.794+0000 7f0520474640 10 monclient(hunting): picked mon.noname-a con 0x7f0518071eb0 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:27.771 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.794+0000 7f0520474640 10 monclient(hunting): picked mon.noname-b con 0x7f0518143d10 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:27.771 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.794+0000 7f0520474640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051806ef80 con 0x7f0518071eb0 2024-03-29T11:21:27.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.794+0000 7f0520474640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051806f260 con 0x7f05180b1f60 2024-03-29T11:21:27.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.794+0000 7f0520474640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05180b40c0 con 0x7f0518143d10 2024-03-29T11:21:27.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.794+0000 7f0520474640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:27.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.794+0000 7f0520474640 10 monclient(hunting): authenticate will time out at 948.090180s 2024-03-29T11:21:27.775 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.798+0000 7f051e210640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f05180b1f60 0x7f0518143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:32989/0 (socket says 172.21.15.67:32989) 2024-03-29T11:21:27.775 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.798+0000 7f051e210640 1 -- 172.21.15.67:0/4192954789 learned_addr learned my addr 172.21.15.67:0/4192954789 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:27.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.800+0000 7f051d20e640 1 -- 172.21.15.67:0/4192954789 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982489422 0 0) 0x7f051806f260 con 0x7f05180b1f60 2024-03-29T11:21:27.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.800+0000 7f051d20e640 10 monclient(hunting): _negotiate 2024-03-29T11:21:27.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.800+0000 7f051d20e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:27.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.800+0000 7f051d20e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:27.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.800+0000 7f051d20e640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:21:27.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.800+0000 7f051d20e640 1 -- 172.21.15.67:0/4192954789 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0504003520 con 0x7f05180b1f60 2024-03-29T11:21:27.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.800+0000 7f051d20e640 1 -- 172.21.15.67:0/4192954789 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1706152688 0 0) 0x7f0504003520 con 0x7f05180b1f60 2024-03-29T11:21:27.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.800+0000 7f051d20e640 1 -- 172.21.15.67:0/4192954789 >> v1:172.21.15.67:6790/0 conn(0x7f0518143d10 legacy=0x7f0518146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.800+0000 7f051d20e640 1 -- 172.21.15.67:0/4192954789 >> v1:172.21.15.67:6789/0 conn(0x7f0518071eb0 legacy=0x7f0518146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.800+0000 7f051d20e640 10 monclient: _finish_hunting 0 2024-03-29T11:21:27.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.800+0000 7f051d20e640 1 monclient: found mon.noname-c 2024-03-29T11:21:27.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.801+0000 7f051d20e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:27.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.801+0000 7f051d20e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:21:27.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.801+0000 7f051d20e640 1 -- 172.21.15.67:0/4192954789 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0518155d70 con 0x7f05180b1f60 2024-03-29T11:21:27.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.801+0000 7f051d20e640 10 monclient: _finish_auth 0 2024-03-29T11:21:27.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.801+0000 7f051d20e640 10 monclient: _check_auth_tickets 2024-03-29T11:21:27.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.801+0000 7f051d20e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:27.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.801+0000 7f051d20e640 1 -- 172.21.15.67:0/4192954789 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0514001690 con 0x7f05180b1f60 2024-03-29T11:21:27.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.801+0000 7f051d20e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:27.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.801+0000 7f051d20e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:27.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.801+0000 7f051d20e640 10 monclient: dump: 2024-03-29T11:21:27.779 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:27.779 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:27.779 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:27.779 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:27.779 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:27.779 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:27.779 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:27.779 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:27.779 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:27.779 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:27.780 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.802+0000 7f0520474640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:21:27.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.802+0000 7f0520474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:27.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.802+0000 7f051d20e640 1 -- 172.21.15.67:0/4192954789 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0514001920 con 0x7f05180b1f60 2024-03-29T11:21:27.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.802+0000 7f051d20e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:27.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.802+0000 7f051d20e640 1 -- 172.21.15.67:0/4192954789 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0514001cb0 con 0x7f05180b1f60 2024-03-29T11:21:27.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.802+0000 7f0520474640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:27.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.802+0000 7f0520474640 4 set_mon_vals no callback set 2024-03-29T11:21:27.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.802+0000 7f051d20e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:27.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.802+0000 7f051d20e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:27.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.802+0000 7f051d20e640 10 monclient: dump: 2024-03-29T11:21:27.781 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:27.782 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:27.782 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:27.782 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:27.782 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:27.782 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:27.782 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:27.782 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:27.782 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:27.782 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:27.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.803+0000 7f0520474640 10 monclient: shutdown 2024-03-29T11:21:27.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.803+0000 7f0520474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:27.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.803+0000 7f0520474640 1 -- 172.21.15.67:0/4192954789 >> v1:172.21.15.82:6789/0 conn(0x7f05180b1f60 legacy=0x7f0518143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.782 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.803+0000 7f0520474640 1 -- 172.21.15.67:0/4192954789 shutdown_connections 2024-03-29T11:21:27.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.803+0000 7f0520474640 1 -- 172.21.15.67:0/4192954789 >> 172.21.15.67:0/4192954789 conn(0x7f0518066a00 msgr2=0x7f0518066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:27.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.803+0000 7f0520474640 1 -- 172.21.15.67:0/4192954789 shutdown_connections 2024-03-29T11:21:27.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.803+0000 7f0520474640 1 -- 172.21.15.67:0/4192954789 wait complete. 2024-03-29T11:21:27.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.804+0000 7f0520474640 10 monclient: build_initial_monmap 2024-03-29T11:21:27.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.804+0000 7f0520474640 10 monclient: monmap: 2024-03-29T11:21:27.785 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:27.785 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:27.785 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:21:27.785 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:21:27.785 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:27.785 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:27.785 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:27.785 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:21:27.785 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:21:27.785 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:27.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.805+0000 7f0520474640 1 librados: starting msgr at 2024-03-29T11:21:27.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.805+0000 7f0520474640 1 librados: starting objecter 2024-03-29T11:21:27.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.805+0000 7f0520474640 1 Processor -- start 2024-03-29T11:21:27.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.805+0000 7f0520474640 1 -- start start 2024-03-29T11:21:27.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.805+0000 7f0520474640 1 librados: setting wanted keys 2024-03-29T11:21:27.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.805+0000 7f0520474640 1 librados: calling monclient init 2024-03-29T11:21:27.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.805+0000 7f0520474640 10 monclient: init 2024-03-29T11:21:27.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.806+0000 7f0520474640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:27.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.806+0000 7f0520474640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:27.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.806+0000 7f0520474640 10 monclient(hunting): picked mon.noname-c con 0x7f05180b1f60 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:27.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.806+0000 7f0520474640 10 monclient(hunting): picked mon.noname-b con 0x7f0518143d10 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:27.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.806+0000 7f0520474640 10 monclient(hunting): picked mon.noname-a con 0x7f0518152850 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:27.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.806+0000 7f0520474640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05181e4a60 con 0x7f0518152850 2024-03-29T11:21:27.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.806+0000 7f0520474640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051814b5b0 con 0x7f0518143d10 2024-03-29T11:21:27.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.806+0000 7f0520474640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05181e4ef0 con 0x7f05180b1f60 2024-03-29T11:21:27.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.806+0000 7f0520474640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:27.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.806+0000 7f0520474640 10 monclient(hunting): authenticate will time out at 948.102095s 2024-03-29T11:21:27.787 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.810+0000 7f051e210640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f05180b1f60 0x7f05181e4100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:40913/0 (socket says 172.21.15.67:40913) 2024-03-29T11:21:27.787 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.810+0000 7f051e210640 1 -- 172.21.15.67:0/1718597317 learned_addr learned my addr 172.21.15.67:0/1718597317 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:27.787 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.811+0000 7f050effd640 10 client.?.objecter ms_handle_connect 0x7f05180b1f60 2024-03-29T11:21:27.787 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.811+0000 7f050effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:27.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.811+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244228194 0 0) 0x7f05181e4ef0 con 0x7f05180b1f60 2024-03-29T11:21:27.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.811+0000 7f050effd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:27.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.811+0000 7f050effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:27.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.811+0000 7f050effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:27.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.811+0000 7f050effd640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:21:27.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.811+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04fc004eb0 con 0x7f05180b1f60 2024-03-29T11:21:27.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 222231161 0 0) 0x7f04fc004eb0 con 0x7f05180b1f60 2024-03-29T11:21:27.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05181e4ef0 con 0x7f05180b1f60 2024-03-29T11:21:27.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0514001650 con 0x7f05180b1f60 2024-03-29T11:21:27.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:27.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:27.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 10 monclient(hunting): dump: 2024-03-29T11:21:27.793 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:27.793 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:27.793 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:27.793 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:27.793 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:27.793 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:27.794 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:27.794 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:27.794 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:27.794 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:27.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:27.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1403528240 0 0) 0x7f05181e4ef0 con 0x7f05180b1f60 2024-03-29T11:21:27.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 >> v1:172.21.15.82:6789/0 conn(0x7f0518143d10 legacy=0x7f0518152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 >> v1:172.21.15.67:6789/0 conn(0x7f0518152850 legacy=0x7f05181491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:27.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 1 monclient: found mon.c 2024-03-29T11:21:27.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:27.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:27.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05181498b0 con 0x7f05180b1f60 2024-03-29T11:21:27.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 10 monclient: _finish_auth 0 2024-03-29T11:21:27.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:27.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:27.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0514004bf0 con 0x7f05180b1f60 2024-03-29T11:21:27.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f050effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:27.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.814+0000 7f051ca0d640 4 set_mon_vals no callback set 2024-03-29T11:21:27.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.815+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0514004f00 con 0x7f05180b1f60 2024-03-29T11:21:27.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.815+0000 7f050effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:27.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.815+0000 7f050effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:27.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.815+0000 7f050effd640 10 monclient: dump: 2024-03-29T11:21:27.795 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:27.795 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:27.795 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:27.795 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:27.796 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:27.796 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:27.796 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:27.796 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:27.796 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:27.796 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:27.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.815+0000 7f0520474640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:21:27.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.815+0000 7f0520474640 10 monclient: _renew_subs 2024-03-29T11:21:27.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.815+0000 7f0520474640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:27.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.815+0000 7f0520474640 1 -- 172.21.15.67:0/1718597317 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0518149b60 con 0x7f05180b1f60 2024-03-29T11:21:27.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.815+0000 7f0520474640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:27.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.815+0000 7f0520474640 10 monclient: _renew_subs 2024-03-29T11:21:27.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.815+0000 7f0520474640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:27.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.815+0000 7f0520474640 1 -- 172.21.15.67:0/1718597317 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0518154db0 con 0x7f05180b1f60 2024-03-29T11:21:27.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.815+0000 7f0520474640 1 librados: init done 2024-03-29T11:21:27.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.819+0000 7f0520474640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:27.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.819+0000 7f0520474640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:27.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.819+0000 7f0520474640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:27.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.819+0000 7f0520474640 1 -- 172.21.15.67:0/1718597317 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05181e4810 con 0x7f05180b1f60 2024-03-29T11:21:27.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.821+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f051401aed0 con 0x7f05180b1f60 2024-03-29T11:21:27.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.821+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f0514084cd0 con 0x7f05180b1f60 2024-03-29T11:21:27.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.821+0000 7f050effd640 10 client.4301.objecter ms_dispatch 0x7f0518000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:27.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.821+0000 7f050effd640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:27.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.821+0000 7f050effd640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:27.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.821+0000 7f050effd640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-29T11:21:27.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.822+0000 7f051da0f640 10 monclient: get_auth_request con 0x7f04fc06dc30 auth_method 0 2024-03-29T11:21:27.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.822+0000 7f050effd640 10 client.4301.objecter ms_handle_connect 0x7f04fc06dc30 2024-03-29T11:21:27.807 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.830+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f05140505b0 con 0x7f05180b1f60 2024-03-29T11:21:27.807 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.830+0000 7f050effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:27.807 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:27.830+0000 7f050effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:28.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.155+0000 7f0520474640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:28.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.155+0000 7f0520474640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:28.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.155+0000 7f0520474640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:28.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.155+0000 7f0520474640 1 -- 172.21.15.67:0/1718597317 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0518075f10 con 0x7f05180b1f60 2024-03-29T11:21:28.133 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.157+0000 7f050effd640 1 -- 172.21.15.67:0/1718597317 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f05140524e0 con 0x7f05180b1f60 2024-03-29T11:21:28.134 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.157+0000 7f050effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:28.134 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.157+0000 7f050effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:28.136 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:21:28.136 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:28.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.161+0000 7f0520474640 10 librados: watch_flush enter 2024-03-29T11:21:28.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.161+0000 7f0520474640 10 librados: watch_flush exit 2024-03-29T11:21:28.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.161+0000 7f0520474640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-29T11:21:28.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.161+0000 7f0520474640 10 client.4301.objecter successfully canceled tick 2024-03-29T11:21:28.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.161+0000 7f0520474640 1 -- 172.21.15.67:0/1718597317 >> v1:172.21.15.67:6816/404787197 conn(0x7f04fc06dc30 legacy=0x7f04fc0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:28.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.161+0000 7f0520474640 10 monclient: shutdown 2024-03-29T11:21:28.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.162+0000 7f0520474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:28.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.162+0000 7f0520474640 1 -- 172.21.15.67:0/1718597317 >> v1:172.21.15.67:6790/0 conn(0x7f05180b1f60 legacy=0x7f05181e4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:28.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.162+0000 7f0520474640 1 -- 172.21.15.67:0/1718597317 shutdown_connections 2024-03-29T11:21:28.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.162+0000 7f0520474640 1 -- 172.21.15.67:0/1718597317 >> 172.21.15.67:0/1718597317 conn(0x7f0518066a00 msgr2=0x7f05180b72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:28.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.162+0000 7f0520474640 1 -- 172.21.15.67:0/1718597317 shutdown_connections 2024-03-29T11:21:28.140 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.162+0000 7f0520474640 1 -- 172.21.15.67:0/1718597317 wait complete. 2024-03-29T11:21:28.140 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:28.162+0000 7f0520474640 1 librados: shutdown 2024-03-29T11:21:28.160 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:34.968 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:35.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.146+0000 7facc2918640 10 monclient: get_monmap_and_config 2024-03-29T11:21:35.124 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.146+0000 7facc2918640 10 monclient: build_initial_monmap 2024-03-29T11:21:35.124 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.146+0000 7facc2918640 10 monclient: monmap: 2024-03-29T11:21:35.124 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:35.124 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:35.124 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:21:35.148239+0000 2024-03-29T11:21:35.124 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:21:35.148239+0000 2024-03-29T11:21:35.124 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:35.124 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:35.124 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:35.124 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:21:35.124 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:21:35.125 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:35.125 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.147+0000 7facc2918640 1 Processor -- start 2024-03-29T11:21:35.128 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.149+0000 7facc2918640 1 -- start start 2024-03-29T11:21:35.128 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.149+0000 7facc2918640 10 monclient: init 2024-03-29T11:21:35.128 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.149+0000 7facc2918640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:35.128 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.149+0000 7facc2918640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:35.128 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.149+0000 7facc2918640 10 monclient(hunting): picked mon.noname-c con 0x7facbc0b2800 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:35.128 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.149+0000 7facc2918640 10 monclient(hunting): picked mon.noname-a con 0x7facbc0a9c00 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:35.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.149+0000 7facc2918640 10 monclient(hunting): picked mon.noname-b con 0x7facbc0b38c0 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:35.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.149+0000 7facc2918640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facbc141eb0 con 0x7facbc0a9c00 2024-03-29T11:21:35.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.149+0000 7facc2918640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facbc0b22d0 con 0x7facbc0b2800 2024-03-29T11:21:35.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.150+0000 7facc2918640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facbc142d30 con 0x7facbc0b38c0 2024-03-29T11:21:35.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.150+0000 7facc2918640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:35.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.150+0000 7facc2918640 10 monclient(hunting): authenticate will time out at 955.446913s 2024-03-29T11:21:35.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.154+0000 7facc1115640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7facbc0a9c00 0x7facbc13bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49731/0 (socket says 172.21.15.67:49731) 2024-03-29T11:21:35.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.154+0000 7facc1115640 1 -- 172.21.15.67:0/89969453 learned_addr learned my addr 172.21.15.67:0/89969453 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:35.133 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.154+0000 7facc0914640 1 -- 172.21.15.67:0/89969453 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786474750 0 0) 0x7facbc141eb0 con 0x7facbc0a9c00 2024-03-29T11:21:35.134 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.154+0000 7facc0914640 10 monclient(hunting): _negotiate 2024-03-29T11:21:35.134 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.154+0000 7facc0914640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:35.134 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.154+0000 7facc0914640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:35.134 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.154+0000 7facc0914640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:21:35.134 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.154+0000 7facc0914640 1 -- 172.21.15.67:0/89969453 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faca8003520 con 0x7facbc0a9c00 2024-03-29T11:21:35.137 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 1 -- 172.21.15.67:0/89969453 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158323179 0 0) 0x7facbc0b22d0 con 0x7facbc0b2800 2024-03-29T11:21:35.137 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 10 monclient(hunting): _negotiate 2024-03-29T11:21:35.137 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:35.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:35.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:21:35.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 1 -- 172.21.15.67:0/89969453 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facbc141eb0 con 0x7facbc0b2800 2024-03-29T11:21:35.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 1 -- 172.21.15.67:0/89969453 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3330384089 0 0) 0x7faca8003520 con 0x7facbc0a9c00 2024-03-29T11:21:35.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 1 -- 172.21.15.67:0/89969453 >> v1:172.21.15.67:6790/0 conn(0x7facbc0b38c0 legacy=0x7facbc14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 1 -- 172.21.15.67:0/89969453 >> v1:172.21.15.82:6789/0 conn(0x7facbc0b2800 legacy=0x7facbc142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 10 monclient: _finish_hunting 0 2024-03-29T11:21:35.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 1 monclient: found mon.noname-a 2024-03-29T11:21:35.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:35.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:21:35.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 1 -- 172.21.15.67:0/89969453 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc0aaf90 con 0x7facbc0a9c00 2024-03-29T11:21:35.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 10 monclient: _finish_auth 0 2024-03-29T11:21:35.138 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 10 monclient: _check_auth_tickets 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 1 -- 172.21.15.67:0/89969453 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7facac001610 con 0x7facbc0a9c00 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.157+0000 7facc0914640 10 monclient: dump: 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:35.139 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:35.140 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.158+0000 7facc0914640 1 -- 172.21.15.67:0/89969453 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facac001770 con 0x7facbc0a9c00 2024-03-29T11:21:35.140 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.158+0000 7facc0914640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:35.140 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.158+0000 7facc0914640 1 -- 172.21.15.67:0/89969453 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7facac001bc0 con 0x7facbc0a9c00 2024-03-29T11:21:35.140 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.158+0000 7facc0914640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:35.140 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.158+0000 7facc0914640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:35.140 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.158+0000 7facc0914640 10 monclient: dump: 2024-03-29T11:21:35.140 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:35.140 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:35.140 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:35.140 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:35.140 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:35.140 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:35.140 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:35.140 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:35.141 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:35.141 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:35.141 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.158+0000 7facc2918640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:21:35.141 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.158+0000 7facc2918640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:35.141 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.158+0000 7facc2918640 4 set_mon_vals no callback set 2024-03-29T11:21:35.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.159+0000 7facc2918640 10 monclient: shutdown 2024-03-29T11:21:35.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.159+0000 7facc2918640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:35.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.159+0000 7facc2918640 1 -- 172.21.15.67:0/89969453 >> v1:172.21.15.67:6789/0 conn(0x7facbc0a9c00 legacy=0x7facbc13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.159+0000 7facc2918640 1 -- 172.21.15.67:0/89969453 shutdown_connections 2024-03-29T11:21:35.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.159+0000 7facc2918640 1 -- 172.21.15.67:0/89969453 >> 172.21.15.67:0/89969453 conn(0x7facbc066a00 msgr2=0x7facbc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:35.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.159+0000 7facc2918640 1 -- 172.21.15.67:0/89969453 shutdown_connections 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.159+0000 7facc2918640 1 -- 172.21.15.67:0/89969453 wait complete. 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.160+0000 7facc2918640 10 monclient: build_initial_monmap 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.160+0000 7facc2918640 10 monclient: monmap: 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.160+0000 7facc2918640 1 librados: starting msgr at 2024-03-29T11:21:35.144 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.160+0000 7facc2918640 1 librados: starting objecter 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.160+0000 7facc2918640 1 Processor -- start 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.161+0000 7facc2918640 1 -- start start 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.161+0000 7facc2918640 1 librados: setting wanted keys 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.161+0000 7facc2918640 1 librados: calling monclient init 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.161+0000 7facc2918640 10 monclient: init 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.161+0000 7facc2918640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.161+0000 7facc2918640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.161+0000 7facc2918640 10 monclient(hunting): picked mon.noname-c con 0x7facbc0a9c00 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.161+0000 7facc2918640 10 monclient(hunting): picked mon.noname-b con 0x7facbc0b2800 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.161+0000 7facc2918640 10 monclient(hunting): picked mon.noname-a con 0x7facbc0b38c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.161+0000 7facc2918640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facbc13b740 con 0x7facbc0b38c0 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.161+0000 7facc2918640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facbc071510 con 0x7facbc0b2800 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.161+0000 7facc2918640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facbc0ab6f0 con 0x7facbc0a9c00 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.161+0000 7facc2918640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:35.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.161+0000 7facc2918640 10 monclient(hunting): authenticate will time out at 955.458819s 2024-03-29T11:21:35.151 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.166+0000 7facc1115640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7facbc0b2800 0x7facbc13a920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:57209/0 (socket says 172.21.15.67:57209) 2024-03-29T11:21:35.151 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.166+0000 7facc1115640 1 -- 172.21.15.67:0/2122023014 learned_addr learned my addr 172.21.15.67:0/2122023014 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:35.151 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.167+0000 7facb27fc640 10 client.?.objecter ms_handle_connect 0x7facbc0b2800 2024-03-29T11:21:35.151 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.167+0000 7facb27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:35.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 10 client.?.objecter ms_handle_connect 0x7facbc0a9c00 2024-03-29T11:21:35.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:35.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261635578 0 0) 0x7facbc0ab6f0 con 0x7facbc0a9c00 2024-03-29T11:21:35.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:35.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:35.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:35.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:21:35.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faca0004eb0 con 0x7facbc0a9c00 2024-03-29T11:21:35.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829197556 0 0) 0x7facbc071510 con 0x7facbc0b2800 2024-03-29T11:21:35.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:35.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:35.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:35.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:21:35.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facbc0ab6f0 con 0x7facbc0b2800 2024-03-29T11:21:35.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1826567946 0 0) 0x7faca0004eb0 con 0x7facbc0a9c00 2024-03-29T11:21:35.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facbc071510 con 0x7facbc0a9c00 2024-03-29T11:21:35.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7facb4001c70 con 0x7facbc0a9c00 2024-03-29T11:21:35.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:35.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:35.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 10 monclient(hunting): dump: 2024-03-29T11:21:35.153 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:35.153 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:35.153 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:35.154 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:35.154 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:35.154 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:35.154 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:35.154 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:35.154 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:35.154 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:35.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.168+0000 7facb27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:35.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.169+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 896070391 0 0) 0x7facbc071510 con 0x7facbc0a9c00 2024-03-29T11:21:35.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.169+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 >> v1:172.21.15.82:6789/0 conn(0x7facbc0b2800 legacy=0x7facbc13a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.169+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 >> v1:172.21.15.67:6789/0 conn(0x7facbc0b38c0 legacy=0x7facbc13b030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:35.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.170+0000 7facb27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:35.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.170+0000 7facb27fc640 1 monclient: found mon.c 2024-03-29T11:21:35.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.170+0000 7facb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:35.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.170+0000 7facb27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:35.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.170+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc070ff0 con 0x7facbc0a9c00 2024-03-29T11:21:35.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.170+0000 7facb27fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:35.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.170+0000 7facb27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:35.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.170+0000 7facb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:35.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.170+0000 7facb27fc640 10 client.?.objecter ms_handle_connect 0x7facbc0b38c0 2024-03-29T11:21:35.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.170+0000 7facb27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:35.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.171+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facb4003990 con 0x7facbc0a9c00 2024-03-29T11:21:35.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.171+0000 7facb27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:35.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.171+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7facb4003ca0 con 0x7facbc0a9c00 2024-03-29T11:21:35.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.171+0000 7facb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:35.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.171+0000 7facb27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:35.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.171+0000 7facb27fc640 10 monclient: dump: 2024-03-29T11:21:35.156 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:35.156 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:35.156 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:35.156 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:35.156 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:35.157 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:35.157 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:35.157 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:35.157 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:35.157 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:35.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.172+0000 7facc2918640 5 monclient: authenticate success, global_id 4307 2024-03-29T11:21:35.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.172+0000 7facc2918640 10 monclient: _renew_subs 2024-03-29T11:21:35.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.172+0000 7facc2918640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:35.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.172+0000 7facc2918640 1 -- 172.21.15.67:0/2122023014 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facbc06ff40 con 0x7facbc0a9c00 2024-03-29T11:21:35.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.172+0000 7facc2918640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:35.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.172+0000 7facc2918640 10 monclient: _renew_subs 2024-03-29T11:21:35.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.172+0000 7facc2918640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:35.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.172+0000 7facc2918640 1 -- 172.21.15.67:0/2122023014 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facbc070c60 con 0x7facbc0a9c00 2024-03-29T11:21:35.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.172+0000 7facc2918640 1 librados: init done 2024-03-29T11:21:35.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.172+0000 7facb37fe640 4 set_mon_vals no callback set 2024-03-29T11:21:35.159 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.175+0000 7facc2918640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:35.159 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.175+0000 7facc2918640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:35.159 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.175+0000 7facc2918640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:35.159 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.175+0000 7facc2918640 1 -- 172.21.15.67:0/2122023014 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca4005080 con 0x7facbc0a9c00 2024-03-29T11:21:35.160 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.177+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7facb4019c70 con 0x7facbc0a9c00 2024-03-29T11:21:35.160 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.177+0000 7facc1115640 10 monclient: get_auth_request con 0x7faca006de10 auth_method 0 2024-03-29T11:21:35.160 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.178+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7facb4083990 con 0x7facbc0a9c00 2024-03-29T11:21:35.160 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.178+0000 7facb27fc640 10 client.4307.objecter ms_dispatch 0x7facbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:35.160 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.178+0000 7facb27fc640 3 client.4307.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:35.160 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.178+0000 7facb27fc640 3 client.4307.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:35.160 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.178+0000 7facb27fc640 20 client.4307.objecter dump_active .. 0 homeless 2024-03-29T11:21:35.160 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.178+0000 7facb27fc640 10 client.4307.objecter ms_handle_connect 0x7faca006de10 2024-03-29T11:21:35.170 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.193+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facb404f3b0 con 0x7facbc0a9c00 2024-03-29T11:21:35.170 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.193+0000 7facb27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:35.170 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.193+0000 7facb27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:35.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.525+0000 7facc2918640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:35.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.525+0000 7facc2918640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:35.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.525+0000 7facc2918640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:35.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.525+0000 7facc2918640 1 -- 172.21.15.67:0/2122023014 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faca4005330 con 0x7facbc0a9c00 2024-03-29T11:21:35.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.527+0000 7facb27fc640 1 -- 172.21.15.67:0/2122023014 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7facb404f3b0 con 0x7facbc0a9c00 2024-03-29T11:21:35.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.527+0000 7facb27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:35.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.527+0000 7facb27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:35.507 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:21:35.508 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:35.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.534+0000 7facc2918640 10 librados: watch_flush enter 2024-03-29T11:21:35.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.534+0000 7facc2918640 10 librados: watch_flush exit 2024-03-29T11:21:35.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.534+0000 7facc2918640 20 client.4307.objecter shutdown clearing up homeless session... 2024-03-29T11:21:35.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.534+0000 7facc2918640 10 client.4307.objecter successfully canceled tick 2024-03-29T11:21:35.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.534+0000 7facc2918640 1 -- 172.21.15.67:0/2122023014 >> v1:172.21.15.67:6816/404787197 conn(0x7faca006de10 legacy=0x7faca00702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.534+0000 7facc2918640 10 monclient: shutdown 2024-03-29T11:21:35.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.534+0000 7facc2918640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:35.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.534+0000 7facc2918640 1 -- 172.21.15.67:0/2122023014 >> v1:172.21.15.67:6790/0 conn(0x7facbc0a9c00 legacy=0x7facbc1d2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.535+0000 7facc2918640 1 -- 172.21.15.67:0/2122023014 shutdown_connections 2024-03-29T11:21:35.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.535+0000 7facc2918640 1 -- 172.21.15.67:0/2122023014 >> 172.21.15.67:0/2122023014 conn(0x7facbc066a00 msgr2=0x7facbc13bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:35.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.535+0000 7facc2918640 1 -- 172.21.15.67:0/2122023014 shutdown_connections 2024-03-29T11:21:35.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.535+0000 7facc2918640 1 -- 172.21.15.67:0/2122023014 wait complete. 2024-03-29T11:21:35.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:35.535+0000 7facc2918640 1 librados: shutdown 2024-03-29T11:21:35.535 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:42.342 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:42.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.528+0000 7f4a06164640 10 monclient: get_monmap_and_config 2024-03-29T11:21:42.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.528+0000 7f4a06164640 10 monclient: build_initial_monmap 2024-03-29T11:21:42.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.528+0000 7f4a06164640 10 monclient: monmap: 2024-03-29T11:21:42.505 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:42.506 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:42.506 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:21:42.529641+0000 2024-03-29T11:21:42.506 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:21:42.529641+0000 2024-03-29T11:21:42.506 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:42.506 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:42.506 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:42.506 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:21:42.506 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:21:42.506 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:42.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.530+0000 7f4a06164640 1 Processor -- start 2024-03-29T11:21:42.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.530+0000 7f4a06164640 1 -- start start 2024-03-29T11:21:42.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.530+0000 7f4a06164640 10 monclient: init 2024-03-29T11:21:42.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.531+0000 7f4a06164640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:42.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.531+0000 7f4a06164640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:42.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.532+0000 7f4a06164640 10 monclient(hunting): picked mon.noname-c con 0x7f49f8006800 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:42.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.532+0000 7f4a06164640 10 monclient(hunting): picked mon.noname-b con 0x7f49f8006be0 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:42.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.532+0000 7f4a06164640 10 monclient(hunting): picked mon.noname-a con 0x7f49f8031b60 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:42.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.532+0000 7f4a06164640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f8029370 con 0x7f49f8031b60 2024-03-29T11:21:42.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.532+0000 7f4a06164640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f80295e0 con 0x7f49f8006800 2024-03-29T11:21:42.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.532+0000 7f4a06164640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f8031360 con 0x7f49f8006be0 2024-03-29T11:21:42.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.532+0000 7f4a06164640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:42.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.532+0000 7f4a06164640 10 monclient(hunting): authenticate will time out at 962.828519s 2024-03-29T11:21:42.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.532+0000 7f4a04961640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f49f8006be0 0x7f49f80c4ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:55320/0 (socket says 172.21.15.67:55320) 2024-03-29T11:21:42.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.532+0000 7f4a04961640 1 -- 172.21.15.67:0/594660154 learned_addr learned my addr 172.21.15.67:0/594660154 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:42.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.533+0000 7f49f7fff640 1 -- 172.21.15.67:0/594660154 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483167200 0 0) 0x7f49f8031360 con 0x7f49f8006be0 2024-03-29T11:21:42.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.533+0000 7f49f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:42.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.533+0000 7f49f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:42.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.533+0000 7f49f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:42.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.533+0000 7f49f7fff640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:21:42.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.533+0000 7f49f7fff640 1 -- 172.21.15.67:0/594660154 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ec003520 con 0x7f49f8006be0 2024-03-29T11:21:42.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.534+0000 7f49f7fff640 1 -- 172.21.15.67:0/594660154 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2726105510 0 0) 0x7f49ec003520 con 0x7f49f8006be0 2024-03-29T11:21:42.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.534+0000 7f49f7fff640 1 -- 172.21.15.67:0/594660154 >> v1:172.21.15.82:6789/0 conn(0x7f49f8006800 legacy=0x7f49f80cad20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:42.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.535+0000 7f49f7fff640 1 -- 172.21.15.67:0/594660154 >> v1:172.21.15.67:6789/0 conn(0x7f49f8031b60 legacy=0x7f49f80d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.535+0000 7f49f7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:42.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.535+0000 7f49f7fff640 1 monclient: found mon.noname-b 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.535+0000 7f49f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.535+0000 7f49f7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.535+0000 7f49f7fff640 1 -- 172.21.15.67:0/594660154 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f8031640 con 0x7f49f8006be0 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.535+0000 7f49f7fff640 10 monclient: _finish_auth 0 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.535+0000 7f49f7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.535+0000 7f49f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.535+0000 7f49f7fff640 1 -- 172.21.15.67:0/594660154 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f49fc001690 con 0x7f49f8006be0 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.535+0000 7f49f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.535+0000 7f49f7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.535+0000 7f49f7fff640 10 monclient: dump: 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:42.515 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:42.516 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:42.516 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:42.516 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:42.516 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:42.516 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:42.516 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.536+0000 7f49f7fff640 1 -- 172.21.15.67:0/594660154 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49fc0017f0 con 0x7f49f8006be0 2024-03-29T11:21:42.516 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.536+0000 7f49f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:42.516 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.536+0000 7f49f7fff640 1 -- 172.21.15.67:0/594660154 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f49fc001c00 con 0x7f49f8006be0 2024-03-29T11:21:42.516 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.536+0000 7f49f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:42.516 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.536+0000 7f49f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:42.516 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.536+0000 7f49f7fff640 10 monclient: dump: 2024-03-29T11:21:42.516 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:42.516 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:42.516 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:42.517 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:42.517 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:42.517 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:42.517 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:42.517 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:42.517 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:42.517 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:42.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.537+0000 7f4a06164640 5 monclient: authenticate success, global_id 4310 2024-03-29T11:21:42.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.537+0000 7f4a06164640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:42.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.537+0000 7f4a06164640 4 set_mon_vals no callback set 2024-03-29T11:21:42.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.538+0000 7f4a06164640 10 monclient: shutdown 2024-03-29T11:21:42.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.538+0000 7f4a06164640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:42.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.538+0000 7f4a06164640 1 -- 172.21.15.67:0/594660154 >> v1:172.21.15.67:6790/0 conn(0x7f49f8006be0 legacy=0x7f49f80c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.519 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.539+0000 7f4a06164640 1 -- 172.21.15.67:0/594660154 shutdown_connections 2024-03-29T11:21:42.519 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.539+0000 7f4a06164640 1 -- 172.21.15.67:0/594660154 >> 172.21.15.67:0/594660154 conn(0x7f49f80058f0 msgr2=0x7f49f8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:42.519 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.540+0000 7f4a06164640 1 -- 172.21.15.67:0/594660154 shutdown_connections 2024-03-29T11:21:42.519 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.541+0000 7f4a06164640 1 -- 172.21.15.67:0/594660154 wait complete. 2024-03-29T11:21:42.519 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.542+0000 7f4a06164640 10 monclient: build_initial_monmap 2024-03-29T11:21:42.519 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.542+0000 7f4a06164640 10 monclient: monmap: 2024-03-29T11:21:42.519 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:42.519 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:42.519 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:21:42.519 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:21:42.519 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:42.520 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:42.520 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:42.520 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:21:42.520 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:21:42.520 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:42.520 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.542+0000 7f4a06164640 1 librados: starting msgr at 2024-03-29T11:21:42.520 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.542+0000 7f4a06164640 1 librados: starting objecter 2024-03-29T11:21:42.520 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.542+0000 7f4a06164640 1 Processor -- start 2024-03-29T11:21:42.520 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.542+0000 7f4a06164640 1 -- start start 2024-03-29T11:21:42.520 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.542+0000 7f4a06164640 1 librados: setting wanted keys 2024-03-29T11:21:42.520 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.543+0000 7f4a06164640 1 librados: calling monclient init 2024-03-29T11:21:42.520 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.543+0000 7f4a06164640 10 monclient: init 2024-03-29T11:21:42.521 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.543+0000 7f4a06164640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:42.521 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.543+0000 7f4a06164640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:42.521 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.543+0000 7f4a06164640 10 monclient(hunting): picked mon.noname-a con 0x7f49f8006800 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:42.521 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.543+0000 7f4a06164640 10 monclient(hunting): picked mon.noname-c con 0x7f49f8031b60 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:42.521 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.544+0000 7f4a04961640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f49f8031b60 0x7f49f815c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:55664/0 (socket says 172.21.15.67:55664) 2024-03-29T11:21:42.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.544+0000 7f4a06164640 10 monclient(hunting): picked mon.noname-b con 0x7f49f80bf910 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:42.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.544+0000 7f4a06164640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f80a2160 con 0x7f49f8006800 2024-03-29T11:21:42.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.544+0000 7f4a06164640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f80a0aa0 con 0x7f49f80bf910 2024-03-29T11:21:42.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.544+0000 7f4a06164640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f815a0d0 con 0x7f49f8031b60 2024-03-29T11:21:42.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.544+0000 7f4a06164640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:42.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.544+0000 7f4a06164640 10 monclient(hunting): authenticate will time out at 962.840705s 2024-03-29T11:21:42.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.545+0000 7f4a05162640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f49f8006800 0x7f49f80bee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33250/0 (socket says 172.21.15.67:33250) 2024-03-29T11:21:42.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.545+0000 7f4a05162640 1 -- 172.21.15.67:0/4231575353 learned_addr learned my addr 172.21.15.67:0/4231575353 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:42.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.545+0000 7f49f5ffb640 10 client.?.objecter ms_handle_connect 0x7f49f8006800 2024-03-29T11:21:42.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.545+0000 7f49f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:42.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.546+0000 7f49f5ffb640 10 client.?.objecter ms_handle_connect 0x7f49f8031b60 2024-03-29T11:21:42.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.546+0000 7f49f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:42.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.546+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223108816 0 0) 0x7f49f80a2160 con 0x7f49f8006800 2024-03-29T11:21:42.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.546+0000 7f49f5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:42.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.546+0000 7f49f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:42.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.546+0000 7f49f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:42.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.546+0000 7f49f5ffb640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:21:42.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.546+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e4004eb0 con 0x7f49f8006800 2024-03-29T11:21:42.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.548+0000 7f49f5ffb640 10 client.?.objecter ms_handle_connect 0x7f49f80bf910 2024-03-29T11:21:42.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.548+0000 7f49f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:42.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.550+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270931953 0 0) 0x7f49f80a0aa0 con 0x7f49f80bf910 2024-03-29T11:21:42.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.550+0000 7f49f5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:42.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.550+0000 7f49f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:42.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.550+0000 7f49f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:42.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.550+0000 7f49f5ffb640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:21:42.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.550+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49f80a2160 con 0x7f49f80bf910 2024-03-29T11:21:42.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.550+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2394890731 0 0) 0x7f49e4004eb0 con 0x7f49f8006800 2024-03-29T11:21:42.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.550+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49f80a0aa0 con 0x7f49f8006800 2024-03-29T11:21:42.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.550+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f49f0001870 con 0x7f49f8006800 2024-03-29T11:21:42.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.550+0000 7f49f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:42.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.550+0000 7f49f5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:42.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.550+0000 7f49f5ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:42.530 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:42.530 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:42.530 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:42.530 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:42.530 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:42.530 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:42.530 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:42.530 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:42.530 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:42.530 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:42.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.550+0000 7f49f5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 60981823 0 0) 0x7f49f80a2160 con 0x7f49f80bf910 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49e4004eb0 con 0x7f49f80bf910 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f4a0004fdd0 con 0x7f49f80bf910 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:42.531 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1164292892 0 0) 0x7f49e4004eb0 con 0x7f49f80bf910 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 >> v1:172.21.15.67:6790/0 conn(0x7f49f8031b60 legacy=0x7f49f815c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 >> v1:172.21.15.67:6789/0 conn(0x7f49f8006800 legacy=0x7f49f80bee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 1 monclient: found mon.b 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f80bf520 con 0x7f49f80bf910 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f49f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f4a06164640 5 monclient: authenticate success, global_id 4294 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f4a06164640 10 monclient: _renew_subs 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f4a06164640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:21:42.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.551+0000 7f4a06164640 1 -- 172.21.15.67:0/4231575353 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f80ca280 con 0x7f49f80bf910 2024-03-29T11:21:42.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.552+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a0004f400 con 0x7f49f80bf910 2024-03-29T11:21:42.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.552+0000 7f4a06164640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:42.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.552+0000 7f49f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:42.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.552+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f4a00075f10 con 0x7f49f80bf910 2024-03-29T11:21:42.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.552+0000 7f49f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:42.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.552+0000 7f49f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:42.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.552+0000 7f49f5ffb640 10 monclient: dump: 2024-03-29T11:21:42.533 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:42.533 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:42.533 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:42.533 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:42.533 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:42.533 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:42.533 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:42.534 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:42.534 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:42.534 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:42.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.553+0000 7f4a06164640 10 monclient: _renew_subs 2024-03-29T11:21:42.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.553+0000 7f4a06164640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:21:42.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.553+0000 7f4a06164640 1 -- 172.21.15.67:0/4231575353 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f8168ec0 con 0x7f49f80bf910 2024-03-29T11:21:42.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.553+0000 7f4a06164640 1 librados: init done 2024-03-29T11:21:42.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.553+0000 7f49f6ffd640 4 set_mon_vals no callback set 2024-03-29T11:21:42.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.555+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f4a00067bf0 con 0x7f49f80bf910 2024-03-29T11:21:42.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.555+0000 7f4a06164640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:42.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.555+0000 7f4a06164640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:42.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.555+0000 7f4a06164640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:21:42.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.555+0000 7f4a06164640 1 -- 172.21.15.67:0/4231575353 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f80a1fa0 con 0x7f49f80bf910 2024-03-29T11:21:42.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.556+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f4a000fe200 con 0x7f49f80bf910 2024-03-29T11:21:42.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.556+0000 7f49f5ffb640 10 client.4294.objecter ms_dispatch 0x7f4a00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:42.536 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.556+0000 7f49f5ffb640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:42.536 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.556+0000 7f49f5ffb640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:42.536 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.556+0000 7f49f5ffb640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-29T11:21:42.536 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.557+0000 7f4a05162640 10 monclient: get_auth_request con 0x7f49e406dde0 auth_method 0 2024-03-29T11:21:42.536 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.558+0000 7f49f5ffb640 10 client.4294.objecter ms_handle_connect 0x7f49e406dde0 2024-03-29T11:21:42.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.562+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a000c9b10 con 0x7f49f80bf910 2024-03-29T11:21:42.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.562+0000 7f49f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:42.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.562+0000 7f49f5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:42.917 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.940+0000 7f4a06164640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:42.917 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.940+0000 7f4a06164640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:42.917 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.940+0000 7f4a06164640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:21:42.917 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.941+0000 7f4a06164640 1 -- 172.21.15.67:0/4231575353 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49f80a0890 con 0x7f49f80bf910 2024-03-29T11:21:42.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.941+0000 7f49f5ffb640 1 -- 172.21.15.67:0/4231575353 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f4a000cba40 con 0x7f49f80bf910 2024-03-29T11:21:42.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.942+0000 7f49f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:42.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.942+0000 7f49f5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:42.918 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:21:42.919 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:42.921 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.945+0000 7f4a06164640 10 librados: watch_flush enter 2024-03-29T11:21:42.921 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.945+0000 7f4a06164640 10 librados: watch_flush exit 2024-03-29T11:21:42.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.945+0000 7f4a06164640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-29T11:21:42.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.945+0000 7f4a06164640 10 client.4294.objecter successfully canceled tick 2024-03-29T11:21:42.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.946+0000 7f4a06164640 1 -- 172.21.15.67:0/4231575353 >> v1:172.21.15.67:6816/404787197 conn(0x7f49e406dde0 legacy=0x7f49e40702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.946+0000 7f4a06164640 10 monclient: shutdown 2024-03-29T11:21:42.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.946+0000 7f4a06164640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:42.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.946+0000 7f4a06164640 1 -- 172.21.15.67:0/4231575353 >> v1:172.21.15.82:6789/0 conn(0x7f49f80bf910 legacy=0x7f49f815cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.923 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.946+0000 7f4a06164640 1 -- 172.21.15.67:0/4231575353 shutdown_connections 2024-03-29T11:21:42.923 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.946+0000 7f4a06164640 1 -- 172.21.15.67:0/4231575353 >> 172.21.15.67:0/4231575353 conn(0x7f49f80058f0 msgr2=0x7f49f802a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:42.923 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.947+0000 7f4a06164640 1 -- 172.21.15.67:0/4231575353 shutdown_connections 2024-03-29T11:21:42.923 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.947+0000 7f4a06164640 1 -- 172.21.15.67:0/4231575353 wait complete. 2024-03-29T11:21:42.923 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:42.947+0000 7f4a06164640 1 librados: shutdown 2024-03-29T11:21:42.941 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:49.748 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:49.879 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.902+0000 7f39fa01b640 10 monclient: get_monmap_and_config 2024-03-29T11:21:49.879 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.902+0000 7f39fa01b640 10 monclient: build_initial_monmap 2024-03-29T11:21:49.879 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.902+0000 7f39fa01b640 10 monclient: monmap: 2024-03-29T11:21:49.879 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:49.879 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:49.879 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:21:49.903640+0000 2024-03-29T11:21:49.879 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:21:49.903640+0000 2024-03-29T11:21:49.879 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:49.879 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:49.879 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:49.879 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:21:49.880 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:21:49.880 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:49.880 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.903+0000 7f39fa01b640 1 Processor -- start 2024-03-29T11:21:49.880 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.903+0000 7f39fa01b640 1 -- start start 2024-03-29T11:21:49.880 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.903+0000 7f39fa01b640 10 monclient: init 2024-03-29T11:21:49.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.904+0000 7f39fa01b640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:49.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.904+0000 7f39fa01b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:49.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.904+0000 7f39fa01b640 10 monclient(hunting): picked mon.noname-c con 0x7f39f40b4820 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:49.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39fa01b640 10 monclient(hunting): picked mon.noname-b con 0x7f39f40a9ce0 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:49.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f9019640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f39f40b4820 0x7f39f413baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:33298/0 (socket says 172.21.15.67:33298) 2024-03-29T11:21:49.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f9019640 1 -- 172.21.15.67:0/4027732899 learned_addr learned my addr 172.21.15.67:0/4027732899 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:49.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f8818640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f39f40a9ce0 0x7f39f4145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:45062/0 (socket says 172.21.15.67:45062) 2024-03-29T11:21:49.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39fa01b640 10 monclient(hunting): picked mon.noname-a con 0x7f39f40b3cd0 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:49.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39fa01b640 1 -- 172.21.15.67:0/4027732899 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f40b4180 con 0x7f39f40b3cd0 2024-03-29T11:21:49.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39fa01b640 1 -- 172.21.15.67:0/4027732899 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f40aaf60 con 0x7f39f40b4820 2024-03-29T11:21:49.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39fa01b640 1 -- 172.21.15.67:0/4027732899 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f413c200 con 0x7f39f40a9ce0 2024-03-29T11:21:49.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39fa01b640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:49.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39fa01b640 10 monclient(hunting): authenticate will time out at 970.201586s 2024-03-29T11:21:49.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 1 -- 172.21.15.67:0/4027732899 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736966635 0 0) 0x7f39f413c200 con 0x7f39f40a9ce0 2024-03-29T11:21:49.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:21:49.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 1 -- 172.21.15.67:0/4027732899 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39e4003520 con 0x7f39f40a9ce0 2024-03-29T11:21:49.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 1 -- 172.21.15.67:0/4027732899 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622372479 0 0) 0x7f39ec001110 con 0x7f39f40b3cd0 2024-03-29T11:21:49.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:21:49.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 1 -- 172.21.15.67:0/4027732899 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f413c200 con 0x7f39f40b3cd0 2024-03-29T11:21:49.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 1 -- 172.21.15.67:0/4027732899 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989642165 0 0) 0x7f39f40aaf60 con 0x7f39f40b4820 2024-03-29T11:21:49.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.905+0000 7f39f3fff640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:21:49.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 1 -- 172.21.15.67:0/4027732899 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39ec001110 con 0x7f39f40b4820 2024-03-29T11:21:49.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 1 -- 172.21.15.67:0/4027732899 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1586989463 0 0) 0x7f39ec001110 con 0x7f39f40b4820 2024-03-29T11:21:49.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 1 -- 172.21.15.67:0/4027732899 >> v1:172.21.15.67:6790/0 conn(0x7f39f40a9ce0 legacy=0x7f39f4145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 1 -- 172.21.15.67:0/4027732899 >> v1:172.21.15.67:6789/0 conn(0x7f39f40b3cd0 legacy=0x7f39f414d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:49.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 1 monclient: found mon.noname-c 2024-03-29T11:21:49.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:49.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:21:49.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 1 -- 172.21.15.67:0/4027732899 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f4145e80 con 0x7f39f40b4820 2024-03-29T11:21:49.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 10 monclient: _finish_auth 0 2024-03-29T11:21:49.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 1 -- 172.21.15.67:0/4027732899 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f39e8001930 con 0x7f39f40b4820 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39f3fff640 10 monclient: dump: 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:49.885 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:49.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39fa01b640 5 monclient: authenticate success, global_id 4297 2024-03-29T11:21:49.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.906+0000 7f39fa01b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:49.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.908+0000 7f39f3fff640 1 -- 172.21.15.67:0/4027732899 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39e8001a90 con 0x7f39f40b4820 2024-03-29T11:21:49.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.908+0000 7f39f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:49.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.908+0000 7f39f3fff640 1 -- 172.21.15.67:0/4027732899 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f39e8001ea0 con 0x7f39f40b4820 2024-03-29T11:21:49.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.908+0000 7f39f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:49.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.908+0000 7f39f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:49.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.908+0000 7f39f3fff640 10 monclient: dump: 2024-03-29T11:21:49.888 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:49.888 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:49.888 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:49.888 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:49.888 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:49.888 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:49.888 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:49.888 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:49.889 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:49.889 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:49.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.908+0000 7f39fa01b640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:49.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.908+0000 7f39fa01b640 4 set_mon_vals no callback set 2024-03-29T11:21:49.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.909+0000 7f39fa01b640 10 monclient: shutdown 2024-03-29T11:21:49.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.909+0000 7f39fa01b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:49.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.909+0000 7f39fa01b640 1 -- 172.21.15.67:0/4027732899 >> v1:172.21.15.82:6789/0 conn(0x7f39f40b4820 legacy=0x7f39f413baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.910+0000 7f39fa01b640 1 -- 172.21.15.67:0/4027732899 shutdown_connections 2024-03-29T11:21:49.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.910+0000 7f39fa01b640 1 -- 172.21.15.67:0/4027732899 >> 172.21.15.67:0/4027732899 conn(0x7f39f406d070 msgr2=0x7f39f4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:49.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.910+0000 7f39fa01b640 1 -- 172.21.15.67:0/4027732899 shutdown_connections 2024-03-29T11:21:49.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.910+0000 7f39fa01b640 1 -- 172.21.15.67:0/4027732899 wait complete. 2024-03-29T11:21:49.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.911+0000 7f39fa01b640 10 monclient: build_initial_monmap 2024-03-29T11:21:49.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.911+0000 7f39fa01b640 10 monclient: monmap: 2024-03-29T11:21:49.890 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:49.890 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:49.890 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:21:49.890 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:21:49.890 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:49.890 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:49.890 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:49.890 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:21:49.890 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:21:49.890 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:49.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.911+0000 7f39fa01b640 1 librados: starting msgr at 2024-03-29T11:21:49.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.911+0000 7f39fa01b640 1 librados: starting objecter 2024-03-29T11:21:49.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.911+0000 7f39fa01b640 1 Processor -- start 2024-03-29T11:21:49.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.911+0000 7f39fa01b640 1 -- start start 2024-03-29T11:21:49.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.911+0000 7f39fa01b640 1 librados: setting wanted keys 2024-03-29T11:21:49.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.911+0000 7f39fa01b640 1 librados: calling monclient init 2024-03-29T11:21:49.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.911+0000 7f39fa01b640 10 monclient: init 2024-03-29T11:21:49.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.912+0000 7f39fa01b640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:49.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.912+0000 7f39fa01b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:49.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.912+0000 7f39fa01b640 10 monclient(hunting): picked mon.noname-c con 0x7f39f40a9ce0 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:49.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.912+0000 7f39fa01b640 10 monclient(hunting): picked mon.noname-b con 0x7f39f40b3cd0 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:49.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.912+0000 7f39fa01b640 10 monclient(hunting): picked mon.noname-a con 0x7f39f40b4820 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:49.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.912+0000 7f39fa01b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f413bca0 con 0x7f39f40b4820 2024-03-29T11:21:49.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.912+0000 7f39fa01b640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f413b9c0 con 0x7f39f40b3cd0 2024-03-29T11:21:49.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.912+0000 7f39fa01b640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f4137f30 con 0x7f39f40a9ce0 2024-03-29T11:21:49.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.912+0000 7f39fa01b640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:49.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.912+0000 7f39fa01b640 10 monclient(hunting): authenticate will time out at 970.208583s 2024-03-29T11:21:49.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.912+0000 7f39f9019640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f39f40a9ce0 0x7f39f41da4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:45306/0 (socket says 172.21.15.67:45306) 2024-03-29T11:21:49.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.912+0000 7f39f9019640 1 -- 172.21.15.67:0/2946776404 learned_addr learned my addr 172.21.15.67:0/2946776404 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:49.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 client.?.objecter ms_handle_connect 0x7f39f40b3cd0 2024-03-29T11:21:49.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:49.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 client.?.objecter ms_handle_connect 0x7f39f40a9ce0 2024-03-29T11:21:49.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:49.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114827707 0 0) 0x7f39f4137f30 con 0x7f39f40a9ce0 2024-03-29T11:21:49.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:21:49.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d8004eb0 con 0x7f39f40a9ce0 2024-03-29T11:21:49.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 client.?.objecter ms_handle_connect 0x7f39f40b4820 2024-03-29T11:21:49.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:49.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305971043 0 0) 0x7f39f413b9c0 con 0x7f39f40b3cd0 2024-03-29T11:21:49.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:21:49.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f4137f30 con 0x7f39f40b3cd0 2024-03-29T11:21:49.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2409905743 0 0) 0x7f39d8004eb0 con 0x7f39f40a9ce0 2024-03-29T11:21:49.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39f413b9c0 con 0x7f39f40a9ce0 2024-03-29T11:21:49.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f39e8001750 con 0x7f39f40a9ce0 2024-03-29T11:21:49.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:49.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:49.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:49.897 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:49.897 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:49.897 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:49.897 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:49.898 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:49.898 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:49.898 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:49.898 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:49.898 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:49.898 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:49.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.913+0000 7f39f1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:49.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186214159 0 0) 0x7f39f413bca0 con 0x7f39f40b4820 2024-03-29T11:21:49.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:21:49.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d8004eb0 con 0x7f39f40b4820 2024-03-29T11:21:49.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3383619370 0 0) 0x7f39f413b9c0 con 0x7f39f40a9ce0 2024-03-29T11:21:49.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 >> v1:172.21.15.82:6789/0 conn(0x7f39f40b3cd0 legacy=0x7f39f413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 >> v1:172.21.15.67:6789/0 conn(0x7f39f40b4820 legacy=0x7f39f4132c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:49.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 1 monclient: found mon.c 2024-03-29T11:21:49.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:49.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:49.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f4137020 con 0x7f39f40a9ce0 2024-03-29T11:21:49.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:49.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:49.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:49.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39fa01b640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:21:49.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39fa01b640 10 monclient: _renew_subs 2024-03-29T11:21:49.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39fa01b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:49.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39fa01b640 1 -- 172.21.15.67:0/2946776404 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f413e580 con 0x7f39f40a9ce0 2024-03-29T11:21:49.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39fa01b640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:49.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39fa01b640 10 monclient: _renew_subs 2024-03-29T11:21:49.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39fa01b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:49.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39fa01b640 1 -- 172.21.15.67:0/2946776404 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f4133570 con 0x7f39f40a9ce0 2024-03-29T11:21:49.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39fa01b640 1 librados: init done 2024-03-29T11:21:49.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39e8004e10 con 0x7f39f40a9ce0 2024-03-29T11:21:49.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:49.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f39e8005120 con 0x7f39f40a9ce0 2024-03-29T11:21:49.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:49.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:49.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f1ffb640 10 monclient: dump: 2024-03-29T11:21:49.900 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:49.900 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.914+0000 7f39f37fe640 4 set_mon_vals no callback set 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.916+0000 7f39fa01b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.916+0000 7f39fa01b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.916+0000 7f39fa01b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.916+0000 7f39fa01b640 1 -- 172.21.15.67:0/2946776404 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d4005080 con 0x7f39f40a9ce0 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.917+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f39e801b0f0 con 0x7f39f40a9ce0 2024-03-29T11:21:49.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.918+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f39e8084dc0 con 0x7f39f40a9ce0 2024-03-29T11:21:49.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.918+0000 7f39f1ffb640 10 client.4319.objecter ms_dispatch 0x7f39f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:49.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.918+0000 7f39f1ffb640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:49.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.918+0000 7f39f1ffb640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:49.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.918+0000 7f39f1ffb640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-29T11:21:49.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.918+0000 7f39f8818640 10 monclient: get_auth_request con 0x7f39d806dea0 auth_method 0 2024-03-29T11:21:49.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.918+0000 7f39f1ffb640 10 client.4319.objecter ms_handle_connect 0x7f39d806dea0 2024-03-29T11:21:49.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.926+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39e80506a0 con 0x7f39f40a9ce0 2024-03-29T11:21:49.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.926+0000 7f39f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:49.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:49.926+0000 7f39f1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:50.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.314+0000 7f39fa01b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:50.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.314+0000 7f39fa01b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:50.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.314+0000 7f39fa01b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:50.291 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.314+0000 7f39fa01b640 1 -- 172.21.15.67:0/2946776404 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39d4005330 con 0x7f39f40a9ce0 2024-03-29T11:21:50.291 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.315+0000 7f39f1ffb640 1 -- 172.21.15.67:0/2946776404 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f39e80525d0 con 0x7f39f40a9ce0 2024-03-29T11:21:50.291 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.315+0000 7f39f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:50.291 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.315+0000 7f39f1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:50.291 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:21:50.292 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:50.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.318+0000 7f39fa01b640 10 librados: watch_flush enter 2024-03-29T11:21:50.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.318+0000 7f39fa01b640 10 librados: watch_flush exit 2024-03-29T11:21:50.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.318+0000 7f39fa01b640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-29T11:21:50.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.318+0000 7f39fa01b640 10 client.4319.objecter successfully canceled tick 2024-03-29T11:21:50.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.318+0000 7f39fa01b640 1 -- 172.21.15.67:0/2946776404 >> v1:172.21.15.67:6816/404787197 conn(0x7f39d806dea0 legacy=0x7f39d8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:50.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.318+0000 7f39fa01b640 10 monclient: shutdown 2024-03-29T11:21:50.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.318+0000 7f39fa01b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:50.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.318+0000 7f39fa01b640 1 -- 172.21.15.67:0/2946776404 >> v1:172.21.15.67:6790/0 conn(0x7f39f40a9ce0 legacy=0x7f39f41da4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:50.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.318+0000 7f39fa01b640 1 -- 172.21.15.67:0/2946776404 shutdown_connections 2024-03-29T11:21:50.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.318+0000 7f39fa01b640 1 -- 172.21.15.67:0/2946776404 >> 172.21.15.67:0/2946776404 conn(0x7f39f406d070 msgr2=0x7f39f40b7280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:50.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.319+0000 7f39fa01b640 1 -- 172.21.15.67:0/2946776404 shutdown_connections 2024-03-29T11:21:50.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.319+0000 7f39fa01b640 1 -- 172.21.15.67:0/2946776404 wait complete. 2024-03-29T11:21:50.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:50.319+0000 7f39fa01b640 1 librados: shutdown 2024-03-29T11:21:50.309 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:57.116 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:57.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.276+0000 7f5a96c86640 10 monclient: get_monmap_and_config 2024-03-29T11:21:57.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.276+0000 7f5a96c86640 10 monclient: build_initial_monmap 2024-03-29T11:21:57.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.276+0000 7f5a96c86640 10 monclient: monmap: 2024-03-29T11:21:57.252 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:57.252 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:57.252 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:21:57.276850+0000 2024-03-29T11:21:57.252 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:21:57.276850+0000 2024-03-29T11:21:57.252 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:57.252 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:57.253 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:57.253 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:21:57.253 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:21:57.253 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:57.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.277+0000 7f5a96c86640 1 Processor -- start 2024-03-29T11:21:57.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.277+0000 7f5a96c86640 1 -- start start 2024-03-29T11:21:57.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.277+0000 7f5a96c86640 10 monclient: init 2024-03-29T11:21:57.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.277+0000 7f5a96c86640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:57.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.277+0000 7f5a96c86640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:57.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.278+0000 7f5a96c86640 10 monclient(hunting): picked mon.noname-b con 0x7f5a9006c250 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:57.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.278+0000 7f5a96c86640 10 monclient(hunting): picked mon.noname-a con 0x7f5a900ae650 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:57.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.278+0000 7f5a96c86640 10 monclient(hunting): picked mon.noname-c con 0x7f5a9013f8c0 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:57.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.278+0000 7f5a96c86640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a900adf50 con 0x7f5a900ae650 2024-03-29T11:21:57.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.278+0000 7f5a96c86640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a900ae230 con 0x7f5a9013f8c0 2024-03-29T11:21:57.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.278+0000 7f5a96c86640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9013fd50 con 0x7f5a9006c250 2024-03-29T11:21:57.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.278+0000 7f5a96c86640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:57.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.278+0000 7f5a96c86640 10 monclient(hunting): authenticate will time out at 977.574064s 2024-03-29T11:21:57.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.278+0000 7f5a95223640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f5a9013f8c0 0x7f5a9014db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:44872/0 (socket says 172.21.15.67:44872) 2024-03-29T11:21:57.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.278+0000 7f5a95223640 1 -- 172.21.15.67:0/1895197119 learned_addr learned my addr 172.21.15.67:0/1895197119 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:57.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.279+0000 7f5a8f7fe640 1 -- 172.21.15.67:0/1895197119 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795600048 0 0) 0x7f5a900ae230 con 0x7f5a9013f8c0 2024-03-29T11:21:57.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.279+0000 7f5a8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:57.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.279+0000 7f5a8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:57.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.279+0000 7f5a8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:57.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.279+0000 7f5a8f7fe640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:21:57.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.279+0000 7f5a8f7fe640 1 -- 172.21.15.67:0/1895197119 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a7c003520 con 0x7f5a9013f8c0 2024-03-29T11:21:57.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.279+0000 7f5a8f7fe640 1 -- 172.21.15.67:0/1895197119 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3417231597 0 0) 0x7f5a7c003520 con 0x7f5a9013f8c0 2024-03-29T11:21:57.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.279+0000 7f5a8f7fe640 1 -- 172.21.15.67:0/1895197119 >> v1:172.21.15.67:6790/0 conn(0x7f5a9006c250 legacy=0x7f5a901427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.279+0000 7f5a8f7fe640 1 -- 172.21.15.67:0/1895197119 >> v1:172.21.15.67:6789/0 conn(0x7f5a900ae650 legacy=0x7f5a9013f1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:57.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.281+0000 7f5a8f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:57.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.281+0000 7f5a8f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:21:57.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.281+0000 7f5a8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:57.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.281+0000 7f5a8f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:21:57.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.281+0000 7f5a8f7fe640 1 -- 172.21.15.67:0/1895197119 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9006db20 con 0x7f5a9013f8c0 2024-03-29T11:21:57.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.281+0000 7f5a8f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:57.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.281+0000 7f5a8f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:57.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.281+0000 7f5a8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:57.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.281+0000 7f5a8f7fe640 1 -- 172.21.15.67:0/1895197119 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5a78001a10 con 0x7f5a9013f8c0 2024-03-29T11:21:57.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.281+0000 7f5a8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:57.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.281+0000 7f5a8f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:57.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.281+0000 7f5a8f7fe640 10 monclient: dump: 2024-03-29T11:21:57.258 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:57.258 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:57.258 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:57.258 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:57.258 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:57.258 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:57.258 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:57.258 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:57.259 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:57.259 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:57.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.281+0000 7f5a96c86640 5 monclient: authenticate success, global_id 4303 2024-03-29T11:21:57.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.281+0000 7f5a96c86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:57.260 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.282+0000 7f5a8f7fe640 1 -- 172.21.15.67:0/1895197119 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a78001c70 con 0x7f5a9013f8c0 2024-03-29T11:21:57.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.282+0000 7f5a8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:57.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.282+0000 7f5a8f7fe640 1 -- 172.21.15.67:0/1895197119 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5a780020c0 con 0x7f5a9013f8c0 2024-03-29T11:21:57.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.282+0000 7f5a8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:57.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.282+0000 7f5a8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:57.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.282+0000 7f5a8f7fe640 10 monclient: dump: 2024-03-29T11:21:57.261 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:57.261 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:57.261 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:57.261 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:57.261 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:57.261 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:57.261 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:57.261 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:57.262 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:57.262 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:57.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.282+0000 7f5a96c86640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:57.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.282+0000 7f5a96c86640 4 set_mon_vals no callback set 2024-03-29T11:21:57.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.283+0000 7f5a96c86640 10 monclient: shutdown 2024-03-29T11:21:57.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.283+0000 7f5a96c86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:57.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.283+0000 7f5a96c86640 1 -- 172.21.15.67:0/1895197119 >> v1:172.21.15.82:6789/0 conn(0x7f5a9013f8c0 legacy=0x7f5a9014db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.283+0000 7f5a96c86640 1 -- 172.21.15.67:0/1895197119 shutdown_connections 2024-03-29T11:21:57.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.283+0000 7f5a96c86640 1 -- 172.21.15.67:0/1895197119 >> 172.21.15.67:0/1895197119 conn(0x7f5a90067610 msgr2=0x7f5a900b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:57.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.283+0000 7f5a96c86640 1 -- 172.21.15.67:0/1895197119 shutdown_connections 2024-03-29T11:21:57.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.283+0000 7f5a96c86640 1 -- 172.21.15.67:0/1895197119 wait complete. 2024-03-29T11:21:57.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.284+0000 7f5a96c86640 10 monclient: build_initial_monmap 2024-03-29T11:21:57.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.284+0000 7f5a96c86640 10 monclient: monmap: 2024-03-29T11:21:57.262 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:21:57.263 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:57.263 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:21:57.263 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:21:57.263 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:57.263 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:57.263 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:21:57.263 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:21:57.263 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:21:57.263 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:57.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.285+0000 7f5a96c86640 1 librados: starting msgr at 2024-03-29T11:21:57.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.285+0000 7f5a96c86640 1 librados: starting objecter 2024-03-29T11:21:57.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.285+0000 7f5a96c86640 1 Processor -- start 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.285+0000 7f5a96c86640 1 -- start start 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.285+0000 7f5a96c86640 1 librados: setting wanted keys 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.285+0000 7f5a96c86640 1 librados: calling monclient init 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.285+0000 7f5a96c86640 10 monclient: init 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.286+0000 7f5a96c86640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.286+0000 7f5a96c86640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.286+0000 7f5a96c86640 10 monclient(hunting): picked mon.noname-b con 0x7f5a9006c250 addr v1:172.21.15.82:6789/0 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.286+0000 7f5a96c86640 10 monclient(hunting): picked mon.noname-c con 0x7f5a900ae650 addr v1:172.21.15.67:6790/0 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.286+0000 7f5a96c86640 10 monclient(hunting): picked mon.noname-a con 0x7f5a9013f8c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.286+0000 7f5a96c86640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9006fce0 con 0x7f5a9013f8c0 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.287+0000 7f5a96c86640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a90072060 con 0x7f5a9006c250 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.287+0000 7f5a96c86640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a90072340 con 0x7f5a900ae650 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.287+0000 7f5a96c86640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.287+0000 7f5a96c86640 10 monclient(hunting): authenticate will time out at 977.582928s 2024-03-29T11:21:57.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.287+0000 7f5a8ffff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f5a900ae650 0x7f5a90139ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:58698/0 (socket says 172.21.15.67:58698) 2024-03-29T11:21:57.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.287+0000 7f5a8ffff640 1 -- 172.21.15.67:0/2987179600 learned_addr learned my addr 172.21.15.67:0/2987179600 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:21:57.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.287+0000 7f5a8d7fa640 10 client.?.objecter ms_handle_connect 0x7f5a900ae650 2024-03-29T11:21:57.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.287+0000 7f5a8d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:57.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136386039 0 0) 0x7f5a90072340 con 0x7f5a900ae650 2024-03-29T11:21:57.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:57.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:57.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:57.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:21:57.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a70004eb0 con 0x7f5a900ae650 2024-03-29T11:21:57.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 client.?.objecter ms_handle_connect 0x7f5a9006c250 2024-03-29T11:21:57.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:57.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140416788 0 0) 0x7f5a90072060 con 0x7f5a9006c250 2024-03-29T11:21:57.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:57.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:57.274 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:57.274 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:21:57.274 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a90072340 con 0x7f5a9006c250 2024-03-29T11:21:57.274 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1675482184 0 0) 0x7f5a70004eb0 con 0x7f5a900ae650 2024-03-29T11:21:57.274 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a90072060 con 0x7f5a900ae650 2024-03-29T11:21:57.274 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5a80001960 con 0x7f5a900ae650 2024-03-29T11:21:57.274 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:57.274 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:57.274 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): dump: 2024-03-29T11:21:57.274 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:57.274 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:57.274 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3107725490 0 0) 0x7f5a90072340 con 0x7f5a9006c250 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a70004eb0 con 0x7f5a9006c250 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5a880018a0 con 0x7f5a9006c250 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 10 monclient(hunting): dump: 2024-03-29T11:21:57.275 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:57.276 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:57.276 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:57.276 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:57.276 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:57.276 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:57.276 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:57.276 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:57.276 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:57.276 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:57.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1454002796 0 0) 0x7f5a90072060 con 0x7f5a900ae650 2024-03-29T11:21:57.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 >> v1:172.21.15.82:6789/0 conn(0x7f5a9006c250 legacy=0x7f5a901deb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.288+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 >> v1:172.21.15.67:6789/0 conn(0x7f5a9013f8c0 legacy=0x7f5a90136c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:57.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a8d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:21:57.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a8d7fa640 1 monclient: found mon.c 2024-03-29T11:21:57.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a8d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:57.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a8d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:57.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a900711c0 con 0x7f5a900ae650 2024-03-29T11:21:57.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a8d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:21:57.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a8d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:21:57.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:57.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a96c86640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:21:57.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a96c86640 10 monclient: _renew_subs 2024-03-29T11:21:57.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a96c86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:57.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a96c86640 1 -- 172.21.15.67:0/2987179600 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a90072760 con 0x7f5a900ae650 2024-03-29T11:21:57.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a96c86640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:57.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a96c86640 10 monclient: _renew_subs 2024-03-29T11:21:57.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a96c86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:57.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a96c86640 1 -- 172.21.15.67:0/2987179600 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a90137560 con 0x7f5a900ae650 2024-03-29T11:21:57.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a96c86640 1 librados: init done 2024-03-29T11:21:57.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a800035f0 con 0x7f5a900ae650 2024-03-29T11:21:57.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a8d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:57.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.289+0000 7f5a8effd640 4 set_mon_vals no callback set 2024-03-29T11:21:57.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.290+0000 7f5a96c86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:57.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.290+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5a80003940 con 0x7f5a900ae650 2024-03-29T11:21:57.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.290+0000 7f5a8d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:57.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.290+0000 7f5a8d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:57.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.290+0000 7f5a8d7fa640 10 monclient: dump: 2024-03-29T11:21:57.278 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:21:57.278 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:21:57.278 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:57.279 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:21:57.279 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:57.279 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:21:57.279 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:21:57.279 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:21:57.279 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:21:57.279 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:21:57.279 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.291+0000 7f5a96c86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:57.279 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.291+0000 7f5a96c86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:57.279 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.291+0000 7f5a96c86640 1 -- 172.21.15.67:0/2987179600 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a74005080 con 0x7f5a900ae650 2024-03-29T11:21:57.279 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.291+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f5a80003b60 con 0x7f5a900ae650 2024-03-29T11:21:57.279 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.291+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f5a800849d0 con 0x7f5a900ae650 2024-03-29T11:21:57.279 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.291+0000 7f5a8d7fa640 10 client.4322.objecter ms_dispatch 0x7f5a90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:57.279 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.291+0000 7f5a8d7fa640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:57.280 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.291+0000 7f5a8d7fa640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:57.280 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.292+0000 7f5a8d7fa640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-29T11:21:57.280 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.292+0000 7f5a94a22640 10 monclient: get_auth_request con 0x7f5a7006df60 auth_method 0 2024-03-29T11:21:57.280 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.292+0000 7f5a8d7fa640 10 client.4322.objecter ms_handle_connect 0x7f5a7006df60 2024-03-29T11:21:57.280 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.299+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a80050370 con 0x7f5a900ae650 2024-03-29T11:21:57.280 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.299+0000 7f5a8d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:57.280 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.299+0000 7f5a8d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:57.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.614+0000 7f5a96c86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:57.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.614+0000 7f5a96c86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:57.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.614+0000 7f5a96c86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:21:57.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.614+0000 7f5a96c86640 1 -- 172.21.15.67:0/2987179600 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a74005330 con 0x7f5a900ae650 2024-03-29T11:21:57.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.615+0000 7f5a8d7fa640 1 -- 172.21.15.67:0/2987179600 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f5a800521e0 con 0x7f5a900ae650 2024-03-29T11:21:57.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.615+0000 7f5a8d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:57.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.615+0000 7f5a8d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:57.592 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:21:57.592 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:57.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.617+0000 7f5a96c86640 10 librados: watch_flush enter 2024-03-29T11:21:57.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.617+0000 7f5a96c86640 10 librados: watch_flush exit 2024-03-29T11:21:57.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.617+0000 7f5a96c86640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-29T11:21:57.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.617+0000 7f5a96c86640 10 client.4322.objecter successfully canceled tick 2024-03-29T11:21:57.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.617+0000 7f5a96c86640 1 -- 172.21.15.67:0/2987179600 >> v1:172.21.15.67:6816/404787197 conn(0x7f5a7006df60 legacy=0x7f5a70070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.617+0000 7f5a96c86640 10 monclient: shutdown 2024-03-29T11:21:57.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.617+0000 7f5a96c86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:57.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.617+0000 7f5a96c86640 1 -- 172.21.15.67:0/2987179600 >> v1:172.21.15.67:6790/0 conn(0x7f5a900ae650 legacy=0x7f5a90139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.618+0000 7f5a96c86640 1 -- 172.21.15.67:0/2987179600 shutdown_connections 2024-03-29T11:21:57.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.618+0000 7f5a96c86640 1 -- 172.21.15.67:0/2987179600 >> 172.21.15.67:0/2987179600 conn(0x7f5a90067610 msgr2=0x7f5a90138090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:57.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.618+0000 7f5a96c86640 1 -- 172.21.15.67:0/2987179600 shutdown_connections 2024-03-29T11:21:57.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.618+0000 7f5a96c86640 1 -- 172.21.15.67:0/2987179600 wait complete. 2024-03-29T11:21:57.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:21:57.618+0000 7f5a96c86640 1 librados: shutdown 2024-03-29T11:21:57.607 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:04.414 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:04.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.593+0000 7fa053a93640 10 monclient: get_monmap_and_config 2024-03-29T11:22:04.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.593+0000 7fa053a93640 10 monclient: build_initial_monmap 2024-03-29T11:22:04.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.593+0000 7fa053a93640 10 monclient: monmap: 2024-03-29T11:22:04.570 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:04.570 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:04.570 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:22:04.594598+0000 2024-03-29T11:22:04.570 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:22:04.594598+0000 2024-03-29T11:22:04.570 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:04.571 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:04.571 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:04.571 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:22:04.571 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:22:04.571 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:04.571 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.595+0000 7fa053a93640 1 Processor -- start 2024-03-29T11:22:04.572 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.595+0000 7fa053a93640 1 -- start start 2024-03-29T11:22:04.572 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.595+0000 7fa053a93640 10 monclient: init 2024-03-29T11:22:04.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.596+0000 7fa053a93640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:04.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.596+0000 7fa053a93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:04.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.596+0000 7fa053a93640 10 monclient(hunting): picked mon.noname-c con 0x7fa04c1457f0 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:04.574 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.596+0000 7fa053a93640 10 monclient(hunting): picked mon.noname-a con 0x7fa04c13f650 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:04.574 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.596+0000 7fa053a93640 10 monclient(hunting): picked mon.noname-b con 0x7fa04c0a9d70 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:04.574 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.596+0000 7fa053a93640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c0aaef0 con 0x7fa04c13f650 2024-03-29T11:22:04.574 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.596+0000 7fa053a93640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c145bd0 con 0x7fa04c1457f0 2024-03-29T11:22:04.574 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.596+0000 7fa053a93640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c145eb0 con 0x7fa04c0a9d70 2024-03-29T11:22:04.574 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.596+0000 7fa053a93640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:04.574 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.596+0000 7fa053a93640 10 monclient(hunting): authenticate will time out at 984.892526s 2024-03-29T11:22:04.576 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.600+0000 7fa052030640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fa04c0a9d70 0x7fa04c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:33694/0 (socket says 172.21.15.67:33694) 2024-03-29T11:22:04.576 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.600+0000 7fa052030640 1 -- 172.21.15.67:0/4014909156 learned_addr learned my addr 172.21.15.67:0/4014909156 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:04.577 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.600+0000 7fa05082d640 1 -- 172.21.15.67:0/4014909156 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520176716 0 0) 0x7fa04c145eb0 con 0x7fa04c0a9d70 2024-03-29T11:22:04.577 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.600+0000 7fa05082d640 10 monclient(hunting): _negotiate 2024-03-29T11:22:04.577 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.600+0000 7fa05082d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:04.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.600+0000 7fa05082d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:04.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.600+0000 7fa05082d640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:22:04.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.600+0000 7fa05082d640 1 -- 172.21.15.67:0/4014909156 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa038003520 con 0x7fa04c0a9d70 2024-03-29T11:22:04.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.601+0000 7fa05082d640 1 -- 172.21.15.67:0/4014909156 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4010835782 0 0) 0x7fa038003520 con 0x7fa04c0a9d70 2024-03-29T11:22:04.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.601+0000 7fa05082d640 1 -- 172.21.15.67:0/4014909156 >> v1:172.21.15.82:6789/0 conn(0x7fa04c1457f0 legacy=0x7fa04c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:04.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.601+0000 7fa05082d640 1 -- 172.21.15.67:0/4014909156 >> v1:172.21.15.67:6789/0 conn(0x7fa04c13f650 legacy=0x7fa04c0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:04.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.602+0000 7fa05082d640 10 monclient: _finish_hunting 0 2024-03-29T11:22:04.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.602+0000 7fa05082d640 1 monclient: found mon.noname-b 2024-03-29T11:22:04.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.602+0000 7fa05082d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:04.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.602+0000 7fa05082d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:22:04.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.602+0000 7fa05082d640 1 -- 172.21.15.67:0/4014909156 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c146340 con 0x7fa04c0a9d70 2024-03-29T11:22:04.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.603+0000 7fa05082d640 10 monclient: _finish_auth 0 2024-03-29T11:22:04.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.603+0000 7fa05082d640 10 monclient: _check_auth_tickets 2024-03-29T11:22:04.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.603+0000 7fa05082d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:04.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.603+0000 7fa05082d640 1 -- 172.21.15.67:0/4014909156 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa044001a10 con 0x7fa04c0a9d70 2024-03-29T11:22:04.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.603+0000 7fa05082d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:04.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.603+0000 7fa05082d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:04.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.603+0000 7fa05082d640 10 monclient: dump: 2024-03-29T11:22:04.583 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:04.583 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:04.583 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:04.583 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:04.583 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:04.583 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:04.584 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:04.584 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:04.584 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:04.584 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:04.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.603+0000 7fa053a93640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:22:04.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.603+0000 7fa053a93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:04.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.604+0000 7fa05082d640 1 -- 172.21.15.67:0/4014909156 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa044001ca0 con 0x7fa04c0a9d70 2024-03-29T11:22:04.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.604+0000 7fa05082d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:04.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.604+0000 7fa05082d640 1 -- 172.21.15.67:0/4014909156 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa044002030 con 0x7fa04c0a9d70 2024-03-29T11:22:04.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.604+0000 7fa05082d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:04.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.604+0000 7fa05082d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:04.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.604+0000 7fa05082d640 10 monclient: dump: 2024-03-29T11:22:04.584 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:04.584 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:04.585 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:04.585 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:04.585 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:04.585 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:04.585 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:04.585 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:04.585 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:04.585 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:04.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.604+0000 7fa053a93640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:04.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.604+0000 7fa053a93640 4 set_mon_vals no callback set 2024-03-29T11:22:04.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.605+0000 7fa053a93640 10 monclient: shutdown 2024-03-29T11:22:04.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.605+0000 7fa053a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:04.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.605+0000 7fa053a93640 1 -- 172.21.15.67:0/4014909156 >> v1:172.21.15.67:6790/0 conn(0x7fa04c0a9d70 legacy=0x7fa04c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:04.588 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.605+0000 7fa053a93640 1 -- 172.21.15.67:0/4014909156 shutdown_connections 2024-03-29T11:22:04.588 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.605+0000 7fa053a93640 1 -- 172.21.15.67:0/4014909156 >> 172.21.15.67:0/4014909156 conn(0x7fa04c066a00 msgr2=0x7fa04c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:04.588 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.608+0000 7fa053a93640 1 -- 172.21.15.67:0/4014909156 shutdown_connections 2024-03-29T11:22:04.588 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.608+0000 7fa053a93640 1 -- 172.21.15.67:0/4014909156 wait complete. 2024-03-29T11:22:04.588 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.609+0000 7fa053a93640 10 monclient: build_initial_monmap 2024-03-29T11:22:04.588 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.609+0000 7fa053a93640 10 monclient: monmap: 2024-03-29T11:22:04.588 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:04.588 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:04.588 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:22:04.588 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:22:04.588 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:04.588 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:04.588 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:04.588 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:22:04.589 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:22:04.589 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:04.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.610+0000 7fa053a93640 1 librados: starting msgr at 2024-03-29T11:22:04.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.610+0000 7fa053a93640 1 librados: starting objecter 2024-03-29T11:22:04.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.610+0000 7fa053a93640 1 Processor -- start 2024-03-29T11:22:04.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.610+0000 7fa053a93640 1 -- start start 2024-03-29T11:22:04.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.610+0000 7fa053a93640 1 librados: setting wanted keys 2024-03-29T11:22:04.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.610+0000 7fa053a93640 1 librados: calling monclient init 2024-03-29T11:22:04.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.610+0000 7fa053a93640 10 monclient: init 2024-03-29T11:22:04.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.611+0000 7fa053a93640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:04.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.611+0000 7fa053a93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:04.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.611+0000 7fa053a93640 10 monclient(hunting): picked mon.noname-a con 0x7fa04c0a9d70 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:04.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.611+0000 7fa053a93640 10 monclient(hunting): picked mon.noname-b con 0x7fa04c13f650 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:04.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.613+0000 7fa053a93640 10 monclient(hunting): picked mon.noname-c con 0x7fa04c1457f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:04.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.613+0000 7fa053a93640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c06f7d0 con 0x7fa04c0a9d70 2024-03-29T11:22:04.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.613+0000 7fa053a93640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c12f730 con 0x7fa04c13f650 2024-03-29T11:22:04.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.613+0000 7fa053a93640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c070910 con 0x7fa04c1457f0 2024-03-29T11:22:04.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.613+0000 7fa053a93640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:04.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.613+0000 7fa053a93640 10 monclient(hunting): authenticate will time out at 984.909323s 2024-03-29T11:22:04.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.615+0000 7fa05102e640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fa04c13f650 0x7fa04c070200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:51463/0 (socket says 172.21.15.67:51463) 2024-03-29T11:22:04.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.615+0000 7fa05102e640 1 -- 172.21.15.67:0/2440299396 learned_addr learned my addr 172.21.15.67:0/2440299396 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:04.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.615+0000 7fa0427fc640 10 client.?.objecter ms_handle_connect 0x7fa04c13f650 2024-03-29T11:22:04.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.615+0000 7fa0427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:04.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855933512 0 0) 0x7fa04c12f730 con 0x7fa04c13f650 2024-03-29T11:22:04.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:04.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:04.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:04.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:22:04.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa030004eb0 con 0x7fa04c13f650 2024-03-29T11:22:04.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1350204315 0 0) 0x7fa030004eb0 con 0x7fa04c13f650 2024-03-29T11:22:04.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa04c12f730 con 0x7fa04c13f650 2024-03-29T11:22:04.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa048001960 con 0x7fa04c13f650 2024-03-29T11:22:04.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:04.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:04.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 10 monclient(hunting): dump: 2024-03-29T11:22:04.597 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:04.597 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:04.597 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:04.597 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:04.597 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:04.597 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:04.597 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:04.597 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:04.597 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:04.597 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:04.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:04.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 10 client.?.objecter ms_handle_connect 0x7fa04c0a9d70 2024-03-29T11:22:04.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.616+0000 7fa0427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:04.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa0427fc640 10 client.?.objecter ms_handle_connect 0x7fa04c1457f0 2024-03-29T11:22:04.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa0427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:04.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2066436328 0 0) 0x7fa04c12f730 con 0x7fa04c13f650 2024-03-29T11:22:04.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 >> v1:172.21.15.67:6790/0 conn(0x7fa04c1457f0 legacy=0x7fa04c1e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:04.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 >> v1:172.21.15.67:6789/0 conn(0x7fa04c0a9d70 legacy=0x7fa04c141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:04.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa0427fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:04.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa0427fc640 1 monclient: found mon.b 2024-03-29T11:22:04.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa0427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:04.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa0427fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:04.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c071e30 con 0x7fa04c13f650 2024-03-29T11:22:04.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa0427fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa0427fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa0427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa053a93640 5 monclient: authenticate success, global_id 4309 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa053a93640 10 monclient: _renew_subs 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa053a93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa053a93640 1 -- 172.21.15.67:0/2440299396 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04c13c720 con 0x7fa04c13f650 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa053a93640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa053a93640 10 monclient: _renew_subs 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa053a93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa053a93640 1 -- 172.21.15.67:0/2440299396 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04c132510 con 0x7fa04c13f650 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.617+0000 7fa053a93640 1 librados: init done 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.618+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa048001c80 con 0x7fa04c13f650 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.618+0000 7fa0427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.618+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa048003a40 con 0x7fa04c13f650 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.618+0000 7fa0427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:04.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.618+0000 7fa0427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.618+0000 7fa0427fc640 10 monclient: dump: 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.618+0000 7fa053a93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.618+0000 7fa053a93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.618+0000 7fa053a93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.618+0000 7fa053a93640 1 -- 172.21.15.67:0/2440299396 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04c071ca0 con 0x7fa04c13f650 2024-03-29T11:22:04.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.618+0000 7fa043fff640 4 set_mon_vals no callback set 2024-03-29T11:22:04.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.620+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fa048003ba0 con 0x7fa04c13f650 2024-03-29T11:22:04.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.621+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fa048084a90 con 0x7fa04c13f650 2024-03-29T11:22:04.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.621+0000 7fa0427fc640 10 client.4309.objecter ms_dispatch 0x7fa04c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:04.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.621+0000 7fa0427fc640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:04.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.621+0000 7fa0427fc640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:04.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.621+0000 7fa0427fc640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-29T11:22:04.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.622+0000 7fa05182f640 10 monclient: get_auth_request con 0x7fa03006dd90 auth_method 0 2024-03-29T11:22:04.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.626+0000 7fa0427fc640 10 client.4309.objecter ms_handle_connect 0x7fa03006dd90 2024-03-29T11:22:04.603 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.626+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa048050370 con 0x7fa04c13f650 2024-03-29T11:22:04.603 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.626+0000 7fa0427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:04.603 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.626+0000 7fa0427fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:04.915 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.939+0000 7fa053a93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:04.915 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.939+0000 7fa053a93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:04.916 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.939+0000 7fa053a93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:04.916 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.939+0000 7fa053a93640 1 -- 172.21.15.67:0/2440299396 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa04c06f980 con 0x7fa04c13f650 2024-03-29T11:22:04.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.942+0000 7fa0427fc640 1 -- 172.21.15.67:0/2440299396 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fa04c06f980 con 0x7fa04c13f650 2024-03-29T11:22:04.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.942+0000 7fa0427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:04.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.942+0000 7fa0427fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:04.918 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:22:04.918 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:04.921 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.945+0000 7fa053a93640 10 librados: watch_flush enter 2024-03-29T11:22:04.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.945+0000 7fa053a93640 10 librados: watch_flush exit 2024-03-29T11:22:04.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.945+0000 7fa053a93640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-29T11:22:04.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.945+0000 7fa053a93640 10 client.4309.objecter successfully canceled tick 2024-03-29T11:22:04.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.945+0000 7fa053a93640 1 -- 172.21.15.67:0/2440299396 >> v1:172.21.15.67:6816/404787197 conn(0x7fa03006dd90 legacy=0x7fa030070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:04.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.945+0000 7fa053a93640 10 monclient: shutdown 2024-03-29T11:22:04.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.945+0000 7fa053a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:04.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.945+0000 7fa053a93640 1 -- 172.21.15.67:0/2440299396 >> v1:172.21.15.82:6789/0 conn(0x7fa04c13f650 legacy=0x7fa04c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:04.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.946+0000 7fa053a93640 1 -- 172.21.15.67:0/2440299396 shutdown_connections 2024-03-29T11:22:04.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.946+0000 7fa053a93640 1 -- 172.21.15.67:0/2440299396 >> 172.21.15.67:0/2440299396 conn(0x7fa04c066a00 msgr2=0x7fa04c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:04.923 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.946+0000 7fa053a93640 1 -- 172.21.15.67:0/2440299396 shutdown_connections 2024-03-29T11:22:04.923 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.946+0000 7fa053a93640 1 -- 172.21.15.67:0/2440299396 wait complete. 2024-03-29T11:22:04.923 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:04.946+0000 7fa053a93640 1 librados: shutdown 2024-03-29T11:22:04.935 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:11.742 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:11.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.910+0000 7fef45c1f640 10 monclient: get_monmap_and_config 2024-03-29T11:22:11.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.910+0000 7fef45c1f640 10 monclient: build_initial_monmap 2024-03-29T11:22:11.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.910+0000 7fef45c1f640 10 monclient: monmap: 2024-03-29T11:22:11.887 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:11.887 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:11.887 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:22:11.911203+0000 2024-03-29T11:22:11.887 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:22:11.911203+0000 2024-03-29T11:22:11.887 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:11.887 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:11.887 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:11.887 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:22:11.887 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:22:11.887 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:11.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.911+0000 7fef45c1f640 1 Processor -- start 2024-03-29T11:22:11.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.911+0000 7fef45c1f640 1 -- start start 2024-03-29T11:22:11.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.911+0000 7fef45c1f640 10 monclient: init 2024-03-29T11:22:11.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.912+0000 7fef45c1f640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:11.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.912+0000 7fef45c1f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:11.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.912+0000 7fef45c1f640 10 monclient(hunting): picked mon.noname-b con 0x7fef400b2800 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:11.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.912+0000 7fef45c1f640 10 monclient(hunting): picked mon.noname-c con 0x7fef400a9c00 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:11.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.913+0000 7fef45c1f640 10 monclient(hunting): picked mon.noname-a con 0x7fef400b38c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:11.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.913+0000 7fef45c1f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef40141eb0 con 0x7fef400b38c0 2024-03-29T11:22:11.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.913+0000 7fef45c1f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef400b22d0 con 0x7fef400a9c00 2024-03-29T11:22:11.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.913+0000 7fef45c1f640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef40142d30 con 0x7fef400b2800 2024-03-29T11:22:11.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.913+0000 7fef45c1f640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:11.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.913+0000 7fef45c1f640 10 monclient(hunting): authenticate will time out at 992.209200s 2024-03-29T11:22:11.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.915+0000 7fef4541e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fef400b38c0 0x7fef4014d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57195/0 (socket says 172.21.15.67:57195) 2024-03-29T11:22:11.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.915+0000 7fef4541e640 1 -- 172.21.15.67:0/1552836206 learned_addr learned my addr 172.21.15.67:0/1552836206 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:11.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.918+0000 7fef3f7fe640 1 -- 172.21.15.67:0/1552836206 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116717691 0 0) 0x7fef40142d30 con 0x7fef400b2800 2024-03-29T11:22:11.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.918+0000 7fef3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.918+0000 7fef3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.918+0000 7fef3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.918+0000 7fef3f7fe640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:22:11.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.918+0000 7fef3f7fe640 1 -- 172.21.15.67:0/1552836206 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef28003520 con 0x7fef400b2800 2024-03-29T11:22:11.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.918+0000 7fef3f7fe640 1 -- 172.21.15.67:0/1552836206 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398680068 0 0) 0x7fef400b22d0 con 0x7fef400a9c00 2024-03-29T11:22:11.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.918+0000 7fef3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.918+0000 7fef3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.918+0000 7fef3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.918+0000 7fef3f7fe640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:22:11.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.918+0000 7fef3f7fe640 1 -- 172.21.15.67:0/1552836206 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef40142d30 con 0x7fef400a9c00 2024-03-29T11:22:11.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 1 -- 172.21.15.67:0/1552836206 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075861007 0 0) 0x7fef40141eb0 con 0x7fef400b38c0 2024-03-29T11:22:11.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:22:11.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 1 -- 172.21.15.67:0/1552836206 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef400b22d0 con 0x7fef400b38c0 2024-03-29T11:22:11.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 1 -- 172.21.15.67:0/1552836206 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1030752869 0 0) 0x7fef40142d30 con 0x7fef400a9c00 2024-03-29T11:22:11.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 1 -- 172.21.15.67:0/1552836206 >> v1:172.21.15.67:6790/0 conn(0x7fef400b2800 legacy=0x7fef40142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 1 -- 172.21.15.67:0/1552836206 >> v1:172.21.15.67:6789/0 conn(0x7fef400b38c0 legacy=0x7fef4014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:11.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:22:11.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:11.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:22:11.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 1 -- 172.21.15.67:0/1552836206 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef400aaf90 con 0x7fef400a9c00 2024-03-29T11:22:11.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:11.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:11.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:11.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 1 -- 172.21.15.67:0/1552836206 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fef30001670 con 0x7fef400a9c00 2024-03-29T11:22:11.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:11.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef3f7fe640 10 monclient: dump: 2024-03-29T11:22:11.899 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:11.899 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:11.899 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:11.899 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:11.899 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.899 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:11.899 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:11.899 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:11.900 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:11.900 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:11.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef45c1f640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:22:11.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.919+0000 7fef45c1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:11.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.922+0000 7fef3f7fe640 1 -- 172.21.15.67:0/1552836206 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef300017d0 con 0x7fef400a9c00 2024-03-29T11:22:11.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.922+0000 7fef3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:11.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.922+0000 7fef3f7fe640 1 -- 172.21.15.67:0/1552836206 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fef30001b60 con 0x7fef400a9c00 2024-03-29T11:22:11.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.922+0000 7fef45c1f640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:11.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.922+0000 7fef45c1f640 4 set_mon_vals no callback set 2024-03-29T11:22:11.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.922+0000 7fef3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.922+0000 7fef3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:11.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.922+0000 7fef3f7fe640 10 monclient: dump: 2024-03-29T11:22:11.902 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:11.902 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:11.902 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:11.902 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:11.902 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.902 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:11.902 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:11.903 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:11.903 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:11.903 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:11.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.923+0000 7fef45c1f640 10 monclient: shutdown 2024-03-29T11:22:11.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.923+0000 7fef45c1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:11.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.923+0000 7fef45c1f640 1 -- 172.21.15.67:0/1552836206 >> v1:172.21.15.82:6789/0 conn(0x7fef400a9c00 legacy=0x7fef4013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.923+0000 7fef45c1f640 1 -- 172.21.15.67:0/1552836206 shutdown_connections 2024-03-29T11:22:11.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.923+0000 7fef45c1f640 1 -- 172.21.15.67:0/1552836206 >> 172.21.15.67:0/1552836206 conn(0x7fef40066a00 msgr2=0x7fef40066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:11.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.923+0000 7fef45c1f640 1 -- 172.21.15.67:0/1552836206 shutdown_connections 2024-03-29T11:22:11.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.923+0000 7fef45c1f640 1 -- 172.21.15.67:0/1552836206 wait complete. 2024-03-29T11:22:11.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.924+0000 7fef45c1f640 10 monclient: build_initial_monmap 2024-03-29T11:22:11.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.924+0000 7fef45c1f640 10 monclient: monmap: 2024-03-29T11:22:11.903 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:11.903 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:11.904 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:22:11.904 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:22:11.904 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:11.904 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:11.904 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:11.904 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:22:11.904 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:22:11.904 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:11.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.925+0000 7fef45c1f640 1 librados: starting msgr at 2024-03-29T11:22:11.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.925+0000 7fef45c1f640 1 librados: starting objecter 2024-03-29T11:22:11.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.925+0000 7fef45c1f640 1 Processor -- start 2024-03-29T11:22:11.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.925+0000 7fef45c1f640 1 -- start start 2024-03-29T11:22:11.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.925+0000 7fef45c1f640 1 librados: setting wanted keys 2024-03-29T11:22:11.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.925+0000 7fef45c1f640 1 librados: calling monclient init 2024-03-29T11:22:11.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.925+0000 7fef45c1f640 10 monclient: init 2024-03-29T11:22:11.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.926+0000 7fef45c1f640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:11.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.926+0000 7fef45c1f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:11.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.926+0000 7fef45c1f640 10 monclient(hunting): picked mon.noname-b con 0x7fef400a9c00 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:11.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef45c1f640 10 monclient(hunting): picked mon.noname-c con 0x7fef400b2800 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:11.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef45c1f640 10 monclient(hunting): picked mon.noname-a con 0x7fef400b38c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:11.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef45c1f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef400af930 con 0x7fef400b38c0 2024-03-29T11:22:11.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef45c1f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef400afc10 con 0x7fef400a9c00 2024-03-29T11:22:11.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef3ffff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fef400b2800 0x7fef400b09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:33320/0 (socket says 172.21.15.67:33320) 2024-03-29T11:22:11.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef3ffff640 1 -- 172.21.15.67:0/2613726098 learned_addr learned my addr 172.21.15.67:0/2613726098 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:11.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef45c1f640 1 -- 172.21.15.67:0/2613726098 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef401d6f40 con 0x7fef400b2800 2024-03-29T11:22:11.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef45c1f640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:11.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef45c1f640 10 monclient(hunting): authenticate will time out at 992.224430s 2024-03-29T11:22:11.912 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef3d7fa640 10 client.?.objecter ms_handle_connect 0x7fef400b2800 2024-03-29T11:22:11.912 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:11.912 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813507898 0 0) 0x7fef401d6f40 con 0x7fef400b2800 2024-03-29T11:22:11.912 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef3d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.912 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.912 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.912 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef3d7fa640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:22:11.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.928+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef2c004eb0 con 0x7fef400b2800 2024-03-29T11:22:11.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.929+0000 7fef3ffff640 1 -- 172.21.15.67:0/2613726098 >> v1:172.21.15.67:6790/0 conn(0x7fef400b2800 legacy=0x7fef400b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T11:22:11.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.929+0000 7fef3ffff640 1 -- 172.21.15.67:0/2613726098 >> v1:172.21.15.67:6790/0 conn(0x7fef400b2800 legacy=0x7fef400b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:22:11.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.929+0000 7fef3ffff640 1 --1- 172.21.15.67:0/2613726098 >> v1:172.21.15.67:6790/0 conn(0x7fef400b2800 0x7fef400b09e0 :-1 s=OPENED pgs=275008 cs=1 l=1).handle_message read tag failed 2024-03-29T11:22:11.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.929+0000 7fef3ffff640 1 --1- 172.21.15.67:0/2613726098 >> v1:172.21.15.67:6790/0 conn(0x7fef400b2800 0x7fef400b09e0 :-1 s=OPENED pgs=275008 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:22:11.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.929+0000 7fef3d7fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.67:6790/0 2024-03-29T11:22:11.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.932+0000 7fef3d7fa640 10 client.?.objecter ms_handle_connect 0x7fef400b38c0 2024-03-29T11:22:11.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.932+0000 7fef3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:11.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.932+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320296466 0 0) 0x7fef400af930 con 0x7fef400b38c0 2024-03-29T11:22:11.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.932+0000 7fef3d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.932+0000 7fef3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.932+0000 7fef3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.932+0000 7fef3d7fa640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:22:11.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.932+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef401d6f40 con 0x7fef400b38c0 2024-03-29T11:22:11.914 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.932+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1858497645 0 0) 0x7fef401d6f40 con 0x7fef400b38c0 2024-03-29T11:22:11.914 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.932+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef400af930 con 0x7fef400b38c0 2024-03-29T11:22:11.914 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.932+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fef38001bd0 con 0x7fef400b38c0 2024-03-29T11:22:11.914 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.933+0000 7fef3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.914 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.933+0000 7fef3d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:11.914 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.933+0000 7fef3d7fa640 10 monclient(hunting): dump: 2024-03-29T11:22:11.914 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:11.914 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:11.914 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:11.914 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:11.914 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.914 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:11.914 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:11.914 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:11.915 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:11.915 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:11.915 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.933+0000 7fef3d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:11.915 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.933+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2340596354 0 0) 0x7fef400af930 con 0x7fef400b38c0 2024-03-29T11:22:11.915 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.933+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 >> v1:172.21.15.67:6790/0 conn(0x7fef400b2800 legacy=0x7fef400b09e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:22:11.915 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.933+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 >> v1:172.21.15.82:6789/0 conn(0x7fef400a9c00 legacy=0x7fef401d7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.915 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.933+0000 7fef3d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:22:11.915 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.933+0000 7fef3d7fa640 1 monclient: found mon.a 2024-03-29T11:22:11.915 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.933+0000 7fef3d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:11.915 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.933+0000 7fef3d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:11.915 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.933+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef401d7b70 con 0x7fef400b38c0 2024-03-29T11:22:11.915 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef3d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:22:11.915 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef3d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:22:11.915 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:11.916 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef45c1f640 5 monclient: authenticate success, global_id 4332 2024-03-29T11:22:11.916 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef38002020 con 0x7fef400b38c0 2024-03-29T11:22:11.916 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef45c1f640 10 monclient: _renew_subs 2024-03-29T11:22:11.916 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef45c1f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:11.916 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef45c1f640 1 -- 172.21.15.67:0/2613726098 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef400b0380 con 0x7fef400b38c0 2024-03-29T11:22:11.916 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:11.916 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fef38004d00 con 0x7fef400b38c0 2024-03-29T11:22:11.916 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.916 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef3d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:11.916 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef3d7fa640 10 monclient: dump: 2024-03-29T11:22:11.916 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:11.916 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:11.916 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:11.916 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:11.917 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.917 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:11.917 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:11.917 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:11.917 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:11.917 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:11.917 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef45c1f640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:11.917 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef45c1f640 10 monclient: _renew_subs 2024-03-29T11:22:11.917 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef45c1f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:11.917 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef45c1f640 1 -- 172.21.15.67:0/2613726098 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef400ae840 con 0x7fef400b38c0 2024-03-29T11:22:11.917 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.934+0000 7fef45c1f640 1 librados: init done 2024-03-29T11:22:11.917 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.936+0000 7fef3effd640 4 set_mon_vals no callback set 2024-03-29T11:22:11.917 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.936+0000 7fef45c1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:11.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.937+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fef3801acd0 con 0x7fef400b38c0 2024-03-29T11:22:11.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.936+0000 7fef45c1f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:11.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.937+0000 7fef45c1f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:11.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.937+0000 7fef45c1f640 1 -- 172.21.15.67:0/2613726098 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef20005080 con 0x7fef400b38c0 2024-03-29T11:22:11.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.938+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fef38084c90 con 0x7fef400b38c0 2024-03-29T11:22:11.919 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.938+0000 7fef3d7fa640 10 client.4332.objecter ms_dispatch 0x7fef40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:11.919 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.938+0000 7fef3d7fa640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:11.919 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.938+0000 7fef3d7fa640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:11.919 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.938+0000 7fef3d7fa640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-29T11:22:11.919 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.939+0000 7fef44c1d640 10 monclient: get_auth_request con 0x7fef2c06dd00 auth_method 0 2024-03-29T11:22:11.919 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.939+0000 7fef3d7fa640 10 client.4332.objecter ms_handle_connect 0x7fef2c06dd00 2024-03-29T11:22:11.923 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.947+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef380503a0 con 0x7fef400b38c0 2024-03-29T11:22:11.923 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.947+0000 7fef3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:11.923 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:11.947+0000 7fef3d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:12.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.271+0000 7fef45c1f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:12.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.271+0000 7fef45c1f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:12.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.271+0000 7fef45c1f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:12.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.271+0000 7fef45c1f640 1 -- 172.21.15.67:0/2613726098 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef20005330 con 0x7fef400b38c0 2024-03-29T11:22:12.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.272+0000 7fef3d7fa640 1 -- 172.21.15.67:0/2613726098 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fef380524a0 con 0x7fef400b38c0 2024-03-29T11:22:12.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.273+0000 7fef3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:12.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.273+0000 7fef3d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:12.249 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:22:12.249 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:12.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.277+0000 7fef45c1f640 10 librados: watch_flush enter 2024-03-29T11:22:12.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.277+0000 7fef45c1f640 10 librados: watch_flush exit 2024-03-29T11:22:12.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.277+0000 7fef45c1f640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-29T11:22:12.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.277+0000 7fef45c1f640 10 client.4332.objecter successfully canceled tick 2024-03-29T11:22:12.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.277+0000 7fef45c1f640 1 -- 172.21.15.67:0/2613726098 >> v1:172.21.15.67:6816/404787197 conn(0x7fef2c06dd00 legacy=0x7fef2c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:12.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.277+0000 7fef45c1f640 10 monclient: shutdown 2024-03-29T11:22:12.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.277+0000 7fef45c1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:12.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.277+0000 7fef45c1f640 1 -- 172.21.15.67:0/2613726098 >> v1:172.21.15.67:6789/0 conn(0x7fef400b38c0 legacy=0x7fef401e27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:12.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.278+0000 7fef45c1f640 1 -- 172.21.15.67:0/2613726098 shutdown_connections 2024-03-29T11:22:12.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.278+0000 7fef45c1f640 1 -- 172.21.15.67:0/2613726098 >> 172.21.15.67:0/2613726098 conn(0x7fef40066a00 msgr2=0x7fef40140450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:12.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.278+0000 7fef45c1f640 1 -- 172.21.15.67:0/2613726098 shutdown_connections 2024-03-29T11:22:12.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.278+0000 7fef45c1f640 1 -- 172.21.15.67:0/2613726098 wait complete. 2024-03-29T11:22:12.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:12.279+0000 7fef45c1f640 1 librados: shutdown 2024-03-29T11:22:12.274 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:19.081 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:19.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.255+0000 7fe81cb40640 10 monclient: get_monmap_and_config 2024-03-29T11:22:19.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.255+0000 7fe81cb40640 10 monclient: build_initial_monmap 2024-03-29T11:22:19.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.255+0000 7fe81cb40640 10 monclient: monmap: 2024-03-29T11:22:19.232 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:19.233 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:19.233 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:22:19.257247+0000 2024-03-29T11:22:19.233 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:22:19.257247+0000 2024-03-29T11:22:19.233 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:19.233 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:19.233 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:19.233 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:22:19.233 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:22:19.233 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:19.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.256+0000 7fe81cb40640 1 Processor -- start 2024-03-29T11:22:19.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.257+0000 7fe81cb40640 1 -- start start 2024-03-29T11:22:19.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.257+0000 7fe81cb40640 10 monclient: init 2024-03-29T11:22:19.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.258+0000 7fe81cb40640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:19.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.258+0000 7fe81cb40640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:19.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.258+0000 7fe81cb40640 10 monclient(hunting): picked mon.noname-b con 0x7fe81806c250 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:19.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.258+0000 7fe81cb40640 10 monclient(hunting): picked mon.noname-a con 0x7fe818072400 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:19.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.258+0000 7fe81cb40640 10 monclient(hunting): picked mon.noname-c con 0x7fe81806c630 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:19.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.258+0000 7fe81cb40640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe81806d760 con 0x7fe818072400 2024-03-29T11:22:19.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.258+0000 7fe81cb40640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe818070500 con 0x7fe81806c630 2024-03-29T11:22:19.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.258+0000 7fe81cb40640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8180aff50 con 0x7fe81806c250 2024-03-29T11:22:19.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.258+0000 7fe81cb40640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:19.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.258+0000 7fe81cb40640 10 monclient(hunting): authenticate will time out at 999.555175s 2024-03-29T11:22:19.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.260+0000 7fe81659c640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fe81806c250 0x7fe818071cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:37216/0 (socket says 172.21.15.67:37216) 2024-03-29T11:22:19.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.260+0000 7fe81659c640 1 -- 172.21.15.67:0/2756591569 learned_addr learned my addr 172.21.15.67:0/2756591569 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:19.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.261+0000 7fe81559a640 1 -- 172.21.15.67:0/2756591569 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241205994 0 0) 0x7fe818070500 con 0x7fe81806c630 2024-03-29T11:22:19.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.262+0000 7fe81559a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:19.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.262+0000 7fe81559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:19.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.262+0000 7fe81559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:19.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.262+0000 7fe81559a640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:22:19.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.262+0000 7fe81559a640 1 -- 172.21.15.67:0/2756591569 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe800003520 con 0x7fe81806c630 2024-03-29T11:22:19.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.262+0000 7fe81559a640 1 -- 172.21.15.67:0/2756591569 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151341262 0 0) 0x7fe8180aff50 con 0x7fe81806c250 2024-03-29T11:22:19.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.262+0000 7fe81559a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:19.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.262+0000 7fe81559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:19.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.262+0000 7fe81559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:19.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.262+0000 7fe81559a640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:22:19.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.262+0000 7fe81559a640 1 -- 172.21.15.67:0/2756591569 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe818070500 con 0x7fe81806c250 2024-03-29T11:22:19.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.262+0000 7fe81559a640 1 -- 172.21.15.67:0/2756591569 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3522059208 0 0) 0x7fe800003520 con 0x7fe81806c630 2024-03-29T11:22:19.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.263+0000 7fe81559a640 1 -- 172.21.15.67:0/2756591569 >> v1:172.21.15.67:6790/0 conn(0x7fe81806c250 legacy=0x7fe818071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.263+0000 7fe81559a640 1 -- 172.21.15.67:0/2756591569 >> v1:172.21.15.67:6789/0 conn(0x7fe818072400 legacy=0x7fe8181465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:19.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.263+0000 7fe81559a640 10 monclient: _finish_hunting 0 2024-03-29T11:22:19.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.263+0000 7fe81559a640 1 monclient: found mon.noname-c 2024-03-29T11:22:19.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.263+0000 7fe81559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:19.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.263+0000 7fe81559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:22:19.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.263+0000 7fe81559a640 1 -- 172.21.15.67:0/2756591569 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8180b00e0 con 0x7fe81806c630 2024-03-29T11:22:19.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.263+0000 7fe81559a640 10 monclient: _finish_auth 0 2024-03-29T11:22:19.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81559a640 10 monclient: _check_auth_tickets 2024-03-29T11:22:19.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:19.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81559a640 1 -- 172.21.15.67:0/2756591569 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe810001a10 con 0x7fe81806c630 2024-03-29T11:22:19.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:19.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:19.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81559a640 10 monclient: dump: 2024-03-29T11:22:19.244 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:19.244 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:19.244 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:19.244 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:19.244 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:19.244 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:19.244 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:19.244 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:19.244 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:19.244 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:19.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81559a640 1 -- 172.21.15.67:0/2756591569 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe810001b70 con 0x7fe81806c630 2024-03-29T11:22:19.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:19.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81559a640 1 -- 172.21.15.67:0/2756591569 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe810001fc0 con 0x7fe81806c630 2024-03-29T11:22:19.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81559a640 10 monclient: dump: 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81cb40640 5 monclient: authenticate success, global_id 4315 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81cb40640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:19.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.264+0000 7fe81cb40640 4 set_mon_vals no callback set 2024-03-29T11:22:19.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.265+0000 7fe81cb40640 10 monclient: shutdown 2024-03-29T11:22:19.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.265+0000 7fe81cb40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:19.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.265+0000 7fe81cb40640 1 -- 172.21.15.67:0/2756591569 >> v1:172.21.15.82:6789/0 conn(0x7fe81806c630 legacy=0x7fe818151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.268+0000 7fe81cb40640 1 -- 172.21.15.67:0/2756591569 shutdown_connections 2024-03-29T11:22:19.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.268+0000 7fe81cb40640 1 -- 172.21.15.67:0/2756591569 >> 172.21.15.67:0/2756591569 conn(0x7fe818067610 msgr2=0x7fe8180b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:19.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.268+0000 7fe81cb40640 1 -- 172.21.15.67:0/2756591569 shutdown_connections 2024-03-29T11:22:19.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.268+0000 7fe81cb40640 1 -- 172.21.15.67:0/2756591569 wait complete. 2024-03-29T11:22:19.251 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.270+0000 7fe81cb40640 10 monclient: build_initial_monmap 2024-03-29T11:22:19.251 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.270+0000 7fe81cb40640 10 monclient: monmap: 2024-03-29T11:22:19.251 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:19.251 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:19.252 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:22:19.252 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:22:19.252 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:19.252 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:19.252 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:19.252 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:22:19.252 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:22:19.252 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:19.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.270+0000 7fe81cb40640 1 librados: starting msgr at 2024-03-29T11:22:19.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.270+0000 7fe81cb40640 1 librados: starting objecter 2024-03-29T11:22:19.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.270+0000 7fe81cb40640 1 Processor -- start 2024-03-29T11:22:19.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.270+0000 7fe81cb40640 1 -- start start 2024-03-29T11:22:19.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.270+0000 7fe81cb40640 1 librados: setting wanted keys 2024-03-29T11:22:19.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.270+0000 7fe81cb40640 1 librados: calling monclient init 2024-03-29T11:22:19.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.270+0000 7fe81cb40640 10 monclient: init 2024-03-29T11:22:19.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.271+0000 7fe81cb40640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:19.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.271+0000 7fe81cb40640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:19.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.271+0000 7fe81cb40640 10 monclient(hunting): picked mon.noname-a con 0x7fe81806c250 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:19.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.271+0000 7fe81cb40640 10 monclient(hunting): picked mon.noname-b con 0x7fe81806c630 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:19.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.271+0000 7fe81cb40640 10 monclient(hunting): picked mon.noname-c con 0x7fe818072400 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:19.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.271+0000 7fe81cb40640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8180aabd0 con 0x7fe81806c250 2024-03-29T11:22:19.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.271+0000 7fe81cb40640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8180ae090 con 0x7fe81806c630 2024-03-29T11:22:19.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.271+0000 7fe81cb40640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8180ae2c0 con 0x7fe818072400 2024-03-29T11:22:19.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.271+0000 7fe81cb40640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:19.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.271+0000 7fe81cb40640 10 monclient(hunting): authenticate will time out at 999.568630s 2024-03-29T11:22:19.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.273+0000 7fe816d9d640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fe818072400 0x7fe8181dbf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:39010/0 (socket says 172.21.15.67:39010) 2024-03-29T11:22:19.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.273+0000 7fe816d9d640 1 -- 172.21.15.67:0/2883423348 learned_addr learned my addr 172.21.15.67:0/2883423348 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:19.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.275+0000 7fe806ffd640 10 client.?.objecter ms_handle_connect 0x7fe818072400 2024-03-29T11:22:19.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.275+0000 7fe806ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:19.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.276+0000 7fe806ffd640 10 client.?.objecter ms_handle_connect 0x7fe81806c630 2024-03-29T11:22:19.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.276+0000 7fe806ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:19.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.276+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528191081 0 0) 0x7fe8180ae2c0 con 0x7fe818072400 2024-03-29T11:22:19.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.276+0000 7fe806ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:19.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.276+0000 7fe806ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:19.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.276+0000 7fe806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:19.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.276+0000 7fe806ffd640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:22:19.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.276+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f8004eb0 con 0x7fe818072400 2024-03-29T11:22:19.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.276+0000 7fe806ffd640 10 client.?.objecter ms_handle_connect 0x7fe81806c250 2024-03-29T11:22:19.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.276+0000 7fe806ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:19.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.277+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2122594732 0 0) 0x7fe7f8004eb0 con 0x7fe818072400 2024-03-29T11:22:19.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.277+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8180ae2c0 con 0x7fe818072400 2024-03-29T11:22:19.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.277+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579762303 0 0) 0x7fe8180ae090 con 0x7fe81806c630 2024-03-29T11:22:19.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.277+0000 7fe806ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:19.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.277+0000 7fe806ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:19.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.277+0000 7fe806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:19.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.277+0000 7fe806ffd640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:22:19.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.277+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f8004eb0 con 0x7fe81806c630 2024-03-29T11:22:19.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.277+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe810001f40 con 0x7fe818072400 2024-03-29T11:22:19.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.277+0000 7fe806ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:19.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.277+0000 7fe806ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:19.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.277+0000 7fe806ffd640 10 monclient(hunting): dump: 2024-03-29T11:22:19.259 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:19.259 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:19.259 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:19.259 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:19.259 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:19.259 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:19.259 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:19.259 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:19.260 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:19.260 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:19.260 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.277+0000 7fe806ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:19.260 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2433904541 0 0) 0x7fe8180ae2c0 con 0x7fe818072400 2024-03-29T11:22:19.260 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 >> v1:172.21.15.82:6789/0 conn(0x7fe81806c630 legacy=0x7fe8180aec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.260 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 >> v1:172.21.15.67:6789/0 conn(0x7fe81806c250 legacy=0x7fe8180acb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.260 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:19.260 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 1 monclient: found mon.c 2024-03-29T11:22:19.260 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:19.260 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:22:19.260 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8180ae6e0 con 0x7fe818072400 2024-03-29T11:22:19.260 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:19.260 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:19.260 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:19.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe810003cc0 con 0x7fe818072400 2024-03-29T11:22:19.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:19.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe810004110 con 0x7fe818072400 2024-03-29T11:22:19.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:19.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:19.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe806ffd640 10 monclient: dump: 2024-03-29T11:22:19.261 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:19.261 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:19.261 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:19.261 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:19.261 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:19.261 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:19.261 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:19.262 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:19.262 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:19.262 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:19.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.278+0000 7fe807fff640 4 set_mon_vals no callback set 2024-03-29T11:22:19.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.280+0000 7fe81cb40640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:22:19.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.280+0000 7fe81cb40640 10 monclient: _renew_subs 2024-03-29T11:22:19.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.280+0000 7fe81cb40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:22:19.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.280+0000 7fe81cb40640 1 -- 172.21.15.67:0/2883423348 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8180af380 con 0x7fe818072400 2024-03-29T11:22:19.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.280+0000 7fe81cb40640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:19.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.280+0000 7fe81cb40640 10 monclient: _renew_subs 2024-03-29T11:22:19.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.280+0000 7fe81cb40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:22:19.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.280+0000 7fe81cb40640 1 -- 172.21.15.67:0/2883423348 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8181dc810 con 0x7fe818072400 2024-03-29T11:22:19.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.280+0000 7fe81cb40640 1 librados: init done 2024-03-29T11:22:19.264 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.282+0000 7fe81cb40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:19.264 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.282+0000 7fe81cb40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:19.264 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.282+0000 7fe81cb40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:22:19.264 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.282+0000 7fe81cb40640 1 -- 172.21.15.67:0/2883423348 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe818075f10 con 0x7fe818072400 2024-03-29T11:22:19.264 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.284+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fe810004270 con 0x7fe818072400 2024-03-29T11:22:19.264 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.284+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fe810084e80 con 0x7fe818072400 2024-03-29T11:22:19.264 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.285+0000 7fe81659c640 10 monclient: get_auth_request con 0x7fe7f806de20 auth_method 0 2024-03-29T11:22:19.264 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.285+0000 7fe806ffd640 10 client.4340.objecter ms_dispatch 0x7fe818000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:19.264 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.285+0000 7fe806ffd640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:19.264 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.285+0000 7fe806ffd640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:19.264 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.285+0000 7fe806ffd640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-29T11:22:19.264 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.285+0000 7fe806ffd640 10 client.4340.objecter ms_handle_connect 0x7fe7f806de20 2024-03-29T11:22:19.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.292+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe810050760 con 0x7fe818072400 2024-03-29T11:22:19.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.292+0000 7fe806ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:19.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.292+0000 7fe806ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:19.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.622+0000 7fe81cb40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:19.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.622+0000 7fe81cb40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:19.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.622+0000 7fe81cb40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:22:19.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.622+0000 7fe81cb40640 1 -- 172.21.15.67:0/2883423348 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8180af510 con 0x7fe818072400 2024-03-29T11:22:19.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.623+0000 7fe806ffd640 1 -- 172.21.15.67:0/2883423348 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fe810052690 con 0x7fe818072400 2024-03-29T11:22:19.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.623+0000 7fe806ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:19.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.623+0000 7fe806ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:19.601 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:22:19.601 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:19.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.627+0000 7fe81cb40640 10 librados: watch_flush enter 2024-03-29T11:22:19.605 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.627+0000 7fe81cb40640 10 librados: watch_flush exit 2024-03-29T11:22:19.605 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.627+0000 7fe81cb40640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-29T11:22:19.605 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.627+0000 7fe81cb40640 10 client.4340.objecter successfully canceled tick 2024-03-29T11:22:19.605 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.627+0000 7fe81cb40640 1 -- 172.21.15.67:0/2883423348 >> v1:172.21.15.67:6816/404787197 conn(0x7fe7f806de20 legacy=0x7fe7f80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.605 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.627+0000 7fe81cb40640 10 monclient: shutdown 2024-03-29T11:22:19.605 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.628+0000 7fe81cb40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:19.605 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.628+0000 7fe81cb40640 1 -- 172.21.15.67:0/2883423348 >> v1:172.21.15.67:6790/0 conn(0x7fe818072400 legacy=0x7fe8181dbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.605 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.628+0000 7fe81cb40640 1 -- 172.21.15.67:0/2883423348 shutdown_connections 2024-03-29T11:22:19.606 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.628+0000 7fe81cb40640 1 -- 172.21.15.67:0/2883423348 >> 172.21.15.67:0/2883423348 conn(0x7fe818067610 msgr2=0x7fe818070920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:19.606 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.628+0000 7fe81cb40640 1 -- 172.21.15.67:0/2883423348 shutdown_connections 2024-03-29T11:22:19.606 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.628+0000 7fe81cb40640 1 -- 172.21.15.67:0/2883423348 wait complete. 2024-03-29T11:22:19.606 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:19.628+0000 7fe81cb40640 1 librados: shutdown 2024-03-29T11:22:19.623 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:26.428 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:26.580 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.603+0000 7fd4c5884640 10 monclient: get_monmap_and_config 2024-03-29T11:22:26.580 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.603+0000 7fd4c5884640 10 monclient: build_initial_monmap 2024-03-29T11:22:26.580 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.603+0000 7fd4c5884640 10 monclient: monmap: 2024-03-29T11:22:26.581 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:26.581 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:26.581 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:22:26.605301+0000 2024-03-29T11:22:26.581 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:22:26.605301+0000 2024-03-29T11:22:26.581 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:26.581 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:26.581 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:26.581 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:22:26.581 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:22:26.581 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:26.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.604+0000 7fd4c5884640 1 Processor -- start 2024-03-29T11:22:26.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.604+0000 7fd4c5884640 1 -- start start 2024-03-29T11:22:26.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.604+0000 7fd4c5884640 10 monclient: init 2024-03-29T11:22:26.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.605+0000 7fd4c5884640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:26.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.605+0000 7fd4c5884640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:26.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.605+0000 7fd4c5884640 10 monclient(hunting): picked mon.noname-b con 0x7fd4c0071de0 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:26.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.605+0000 7fd4c5884640 10 monclient(hunting): picked mon.noname-a con 0x7fd4c00721c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:26.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.605+0000 7fd4c5884640 10 monclient(hunting): picked mon.noname-c con 0x7fd4c0145ae0 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:26.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.605+0000 7fd4c5884640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c0145650 con 0x7fd4c00721c0 2024-03-29T11:22:26.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.605+0000 7fd4c5884640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c00ae260 con 0x7fd4c0145ae0 2024-03-29T11:22:26.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.605+0000 7fd4c5884640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c01420d0 con 0x7fd4c0071de0 2024-03-29T11:22:26.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.605+0000 7fd4c5884640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:26.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.605+0000 7fd4c5884640 10 monclient(hunting): authenticate will time out at 1006.902509s 2024-03-29T11:22:26.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.609+0000 7fd4beffd640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fd4c0071de0 0x7fd4c00aed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:42379/0 (socket says 172.21.15.67:42379) 2024-03-29T11:22:26.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.609+0000 7fd4beffd640 1 -- 172.21.15.67:0/730389264 learned_addr learned my addr 172.21.15.67:0/730389264 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:26.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.609+0000 7fd4bdffb640 1 -- 172.21.15.67:0/730389264 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272744177 0 0) 0x7fd4c01420d0 con 0x7fd4c0071de0 2024-03-29T11:22:26.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.609+0000 7fd4bdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:26.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.609+0000 7fd4bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:26.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.609+0000 7fd4bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:26.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.609+0000 7fd4bdffb640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:22:26.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.609+0000 7fd4bdffb640 1 -- 172.21.15.67:0/730389264 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4a8003520 con 0x7fd4c0071de0 2024-03-29T11:22:26.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.612+0000 7fd4bdffb640 1 -- 172.21.15.67:0/730389264 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3736816928 0 0) 0x7fd4a8003520 con 0x7fd4c0071de0 2024-03-29T11:22:26.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.612+0000 7fd4bdffb640 1 -- 172.21.15.67:0/730389264 >> v1:172.21.15.82:6789/0 conn(0x7fd4c0145ae0 legacy=0x7fd4c0145f50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:22:26.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.612+0000 7fd4bdffb640 1 -- 172.21.15.67:0/730389264 >> v1:172.21.15.67:6789/0 conn(0x7fd4c00721c0 legacy=0x7fd4c013f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:26.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:26.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 1 monclient: found mon.noname-b 2024-03-29T11:22:26.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:26.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:22:26.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 1 -- 172.21.15.67:0/730389264 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c00b0590 con 0x7fd4c0071de0 2024-03-29T11:22:26.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:26.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:26.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:26.593 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 1 -- 172.21.15.67:0/730389264 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd4b0001690 con 0x7fd4c0071de0 2024-03-29T11:22:26.593 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:26.593 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:26.593 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 10 monclient: dump: 2024-03-29T11:22:26.593 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:26.593 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:26.593 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:26.593 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:26.593 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:26.593 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:26.593 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:26.593 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:26.593 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:26.593 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:26.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4c5884640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:22:26.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4c5884640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:26.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 1 -- 172.21.15.67:0/730389264 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4b00017f0 con 0x7fd4c0071de0 2024-03-29T11:22:26.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:26.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 1 -- 172.21.15.67:0/730389264 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd4b0001c40 con 0x7fd4c0071de0 2024-03-29T11:22:26.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:26.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:26.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4bdffb640 10 monclient: dump: 2024-03-29T11:22:26.594 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:26.594 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:26.594 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:26.594 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:26.594 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:26.595 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:26.595 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:26.595 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:26.595 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:26.595 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:26.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4c5884640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:26.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.613+0000 7fd4c5884640 4 set_mon_vals no callback set 2024-03-29T11:22:26.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.614+0000 7fd4c5884640 10 monclient: shutdown 2024-03-29T11:22:26.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.614+0000 7fd4c5884640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:26.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.614+0000 7fd4c5884640 1 -- 172.21.15.67:0/730389264 >> v1:172.21.15.67:6790/0 conn(0x7fd4c0071de0 legacy=0x7fd4c00aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.615+0000 7fd4c5884640 1 -- 172.21.15.67:0/730389264 shutdown_connections 2024-03-29T11:22:26.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.615+0000 7fd4c5884640 1 -- 172.21.15.67:0/730389264 >> 172.21.15.67:0/730389264 conn(0x7fd4c006b660 msgr2=0x7fd4c006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:26.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.618+0000 7fd4c5884640 1 -- 172.21.15.67:0/730389264 shutdown_connections 2024-03-29T11:22:26.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.618+0000 7fd4c5884640 1 -- 172.21.15.67:0/730389264 wait complete. 2024-03-29T11:22:26.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.619+0000 7fd4c5884640 10 monclient: build_initial_monmap 2024-03-29T11:22:26.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.619+0000 7fd4c5884640 10 monclient: monmap: 2024-03-29T11:22:26.596 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:26.596 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:26.596 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:22:26.596 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:22:26.597 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:26.597 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:26.597 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:26.597 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:22:26.597 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:22:26.597 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:26.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.619+0000 7fd4c5884640 1 librados: starting msgr at 2024-03-29T11:22:26.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.619+0000 7fd4c5884640 1 librados: starting objecter 2024-03-29T11:22:26.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.619+0000 7fd4c5884640 1 Processor -- start 2024-03-29T11:22:26.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.620+0000 7fd4c5884640 1 -- start start 2024-03-29T11:22:26.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.620+0000 7fd4c5884640 1 librados: setting wanted keys 2024-03-29T11:22:26.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.620+0000 7fd4c5884640 1 librados: calling monclient init 2024-03-29T11:22:26.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.620+0000 7fd4c5884640 10 monclient: init 2024-03-29T11:22:26.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.622+0000 7fd4c5884640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:26.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.622+0000 7fd4c5884640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:26.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.623+0000 7fd4c5884640 10 monclient(hunting): picked mon.noname-b con 0x7fd4c0071de0 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:26.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.623+0000 7fd4c5884640 10 monclient(hunting): picked mon.noname-c con 0x7fd4c0145ae0 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:26.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.623+0000 7fd4c5884640 10 monclient(hunting): picked mon.noname-a con 0x7fd4c01318e0 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:26.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.623+0000 7fd4c5884640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c0144ef0 con 0x7fd4c01318e0 2024-03-29T11:22:26.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.623+0000 7fd4c5884640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c0144be0 con 0x7fd4c0071de0 2024-03-29T11:22:26.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.623+0000 7fd4c5884640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c012fed0 con 0x7fd4c0145ae0 2024-03-29T11:22:26.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.623+0000 7fd4c5884640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:26.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.623+0000 7fd4c5884640 10 monclient(hunting): authenticate will time out at 1006.919999s 2024-03-29T11:22:26.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.626+0000 7fd4beffd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fd4c0071de0 0x7fd4c012f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:44931/0 (socket says 172.21.15.67:44931) 2024-03-29T11:22:26.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.626+0000 7fd4beffd640 1 -- 172.21.15.67:0/3380222320 learned_addr learned my addr 172.21.15.67:0/3380222320 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:26.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.626+0000 7fd4a7fff640 10 client.?.objecter ms_handle_connect 0x7fd4c0071de0 2024-03-29T11:22:26.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.626+0000 7fd4a7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:26.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.626+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295632320 0 0) 0x7fd4c0144be0 con 0x7fd4c0071de0 2024-03-29T11:22:26.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.626+0000 7fd4a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:26.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.626+0000 7fd4a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:26.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.626+0000 7fd4a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:26.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.626+0000 7fd4a7fff640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:22:26.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.626+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4ac004eb0 con 0x7fd4c0071de0 2024-03-29T11:22:26.605 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.627+0000 7fd4a7fff640 10 client.?.objecter ms_handle_connect 0x7fd4c0145ae0 2024-03-29T11:22:26.605 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.627+0000 7fd4a7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:26.605 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.628+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903359838 0 0) 0x7fd4c012fed0 con 0x7fd4c0145ae0 2024-03-29T11:22:26.607 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.628+0000 7fd4a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:26.607 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.628+0000 7fd4a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:26.607 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.628+0000 7fd4a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:26.607 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.628+0000 7fd4a7fff640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:22:26.607 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.628+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c0144be0 con 0x7fd4c0145ae0 2024-03-29T11:22:26.607 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.628+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3475709384 0 0) 0x7fd4ac004eb0 con 0x7fd4c0071de0 2024-03-29T11:22:26.607 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.628+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4ac005490 con 0x7fd4c0071de0 2024-03-29T11:22:26.607 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.628+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd4b0001c00 con 0x7fd4c0071de0 2024-03-29T11:22:26.607 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.628+0000 7fd4a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:26.607 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.628+0000 7fd4a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:26.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.628+0000 7fd4a7fff640 10 monclient(hunting): dump: 2024-03-29T11:22:26.608 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:26.608 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:26.608 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:26.608 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:26.608 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:26.608 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:26.608 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:26.608 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:26.608 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:26.608 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:26.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.628+0000 7fd4a7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:26.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.629+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 762736631 0 0) 0x7fd4ac005490 con 0x7fd4c0071de0 2024-03-29T11:22:26.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.629+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 >> v1:172.21.15.67:6790/0 conn(0x7fd4c0145ae0 legacy=0x7fd4c01da620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.629+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 >> v1:172.21.15.67:6789/0 conn(0x7fd4c01318e0 legacy=0x7fd4c01dad30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:26.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4a7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:26.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4a7fff640 1 monclient: found mon.b 2024-03-29T11:22:26.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:26.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4a7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:26.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c0130060 con 0x7fd4c0071de0 2024-03-29T11:22:26.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4a7fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:26.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4a7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:26.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:26.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4c5884640 5 monclient: authenticate success, global_id 4321 2024-03-29T11:22:26.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4c5884640 10 monclient: _renew_subs 2024-03-29T11:22:26.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4c5884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:26.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4c5884640 1 -- 172.21.15.67:0/3380222320 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c00ad100 con 0x7fd4c0071de0 2024-03-29T11:22:26.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4c5884640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:26.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4c5884640 10 monclient: _renew_subs 2024-03-29T11:22:26.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4c5884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:26.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4c5884640 1 -- 172.21.15.67:0/3380222320 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c00ac3b0 con 0x7fd4c0071de0 2024-03-29T11:22:26.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4c5884640 1 librados: init done 2024-03-29T11:22:26.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4b0004a90 con 0x7fd4c0071de0 2024-03-29T11:22:26.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:26.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd4b0004da0 con 0x7fd4c0071de0 2024-03-29T11:22:26.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:26.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:26.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4a7fff640 10 monclient: dump: 2024-03-29T11:22:26.611 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:26.612 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:26.612 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:26.612 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:26.612 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:26.612 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:26.612 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:26.612 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:26.612 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:26.612 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:26.612 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.632+0000 7fd4bd7fa640 4 set_mon_vals no callback set 2024-03-29T11:22:26.612 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.634+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fd4b001ad70 con 0x7fd4c0071de0 2024-03-29T11:22:26.613 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.634+0000 7fd4c5884640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:26.613 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.635+0000 7fd4c5884640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:26.613 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.635+0000 7fd4c5884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:26.613 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.635+0000 7fd4c5884640 1 -- 172.21.15.67:0/3380222320 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a0005080 con 0x7fd4c0071de0 2024-03-29T11:22:26.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.635+0000 7fd4be7fc640 10 monclient: get_auth_request con 0x7fd4ac06df30 auth_method 0 2024-03-29T11:22:26.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.635+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fd4b0084bd0 con 0x7fd4c0071de0 2024-03-29T11:22:26.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.635+0000 7fd4a7fff640 10 client.4321.objecter ms_dispatch 0x7fd4c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:26.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.635+0000 7fd4a7fff640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:26.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.635+0000 7fd4a7fff640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:26.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.635+0000 7fd4a7fff640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-29T11:22:26.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.635+0000 7fd4a7fff640 10 client.4321.objecter ms_handle_connect 0x7fd4ac06df30 2024-03-29T11:22:26.620 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.643+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd4b00504b0 con 0x7fd4c0071de0 2024-03-29T11:22:26.620 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.643+0000 7fd4a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:26.620 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.643+0000 7fd4a7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:26.945 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.968+0000 7fd4c5884640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:26.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.968+0000 7fd4c5884640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:26.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.968+0000 7fd4c5884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:26.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.968+0000 7fd4c5884640 1 -- 172.21.15.67:0/3380222320 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4a0005330 con 0x7fd4c0071de0 2024-03-29T11:22:26.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.972+0000 7fd4a7fff640 1 -- 172.21.15.67:0/3380222320 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fd4b00523e0 con 0x7fd4c0071de0 2024-03-29T11:22:26.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.972+0000 7fd4a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:26.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.972+0000 7fd4a7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:26.952 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:22:26.953 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:26.956 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.978+0000 7fd4c5884640 10 librados: watch_flush enter 2024-03-29T11:22:26.956 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.978+0000 7fd4c5884640 10 librados: watch_flush exit 2024-03-29T11:22:26.956 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.978+0000 7fd4c5884640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-29T11:22:26.956 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.978+0000 7fd4c5884640 10 client.4321.objecter successfully canceled tick 2024-03-29T11:22:26.956 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.979+0000 7fd4c5884640 1 -- 172.21.15.67:0/3380222320 >> v1:172.21.15.67:6816/404787197 conn(0x7fd4ac06df30 legacy=0x7fd4ac0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.956 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.979+0000 7fd4c5884640 10 monclient: shutdown 2024-03-29T11:22:26.956 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.979+0000 7fd4c5884640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:26.957 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.979+0000 7fd4c5884640 1 -- 172.21.15.67:0/3380222320 >> v1:172.21.15.82:6789/0 conn(0x7fd4c0071de0 legacy=0x7fd4c012f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.957 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.979+0000 7fd4c5884640 1 -- 172.21.15.67:0/3380222320 shutdown_connections 2024-03-29T11:22:26.957 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.979+0000 7fd4c5884640 1 -- 172.21.15.67:0/3380222320 >> 172.21.15.67:0/3380222320 conn(0x7fd4c006b660 msgr2=0x7fd4c00b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:26.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.982+0000 7fd4c5884640 1 -- 172.21.15.67:0/3380222320 shutdown_connections 2024-03-29T11:22:26.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.982+0000 7fd4c5884640 1 -- 172.21.15.67:0/3380222320 wait complete. 2024-03-29T11:22:26.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:26.982+0000 7fd4c5884640 1 librados: shutdown 2024-03-29T11:22:26.977 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:33.784 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:33.938 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.960+0000 7fc5e950f640 10 monclient: get_monmap_and_config 2024-03-29T11:22:33.938 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.960+0000 7fc5e950f640 10 monclient: build_initial_monmap 2024-03-29T11:22:33.938 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.960+0000 7fc5e950f640 10 monclient: monmap: 2024-03-29T11:22:33.938 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:33.938 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:33.939 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:22:33.961171+0000 2024-03-29T11:22:33.939 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:22:33.961171+0000 2024-03-29T11:22:33.939 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:33.939 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:33.939 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:33.939 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:22:33.939 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:22:33.939 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:33.943 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.963+0000 7fc5e950f640 1 Processor -- start 2024-03-29T11:22:33.945 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.963+0000 7fc5e950f640 1 -- start start 2024-03-29T11:22:33.945 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.963+0000 7fc5e950f640 10 monclient: init 2024-03-29T11:22:33.945 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.964+0000 7fc5e950f640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:33.945 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.964+0000 7fc5e950f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:33.945 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.964+0000 7fc5e950f640 10 monclient(hunting): picked mon.noname-c con 0x7fc5e40a9930 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:33.945 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.964+0000 7fc5e950f640 10 monclient(hunting): picked mon.noname-b con 0x7fc5e4149dc0 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:33.945 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.964+0000 7fc5e950f640 10 monclient(hunting): picked mon.noname-a con 0x7fc5e414ff40 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:33.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.964+0000 7fc5e950f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e40aae60 con 0x7fc5e414ff40 2024-03-29T11:22:33.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.964+0000 7fc5e950f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e40b4310 con 0x7fc5e40a9930 2024-03-29T11:22:33.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.964+0000 7fc5e950f640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e40afdd0 con 0x7fc5e4149dc0 2024-03-29T11:22:33.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.964+0000 7fc5e950f640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:33.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.964+0000 7fc5e950f640 10 monclient(hunting): authenticate will time out at 1014.260127s 2024-03-29T11:22:33.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.967+0000 7fc5e2ffd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fc5e40a9930 0x7fc5e4143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:58141/0 (socket says 172.21.15.67:58141) 2024-03-29T11:22:33.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.967+0000 7fc5e2ffd640 1 -- 172.21.15.67:0/3707839411 learned_addr learned my addr 172.21.15.67:0/3707839411 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:33.948 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.971+0000 7fc5e1ffb640 1 -- 172.21.15.67:0/3707839411 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697809783 0 0) 0x7fc5e40aae60 con 0x7fc5e414ff40 2024-03-29T11:22:33.948 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:33.948 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:33.948 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:33.948 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:22:33.948 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 1 -- 172.21.15.67:0/3707839411 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5cc003520 con 0x7fc5e414ff40 2024-03-29T11:22:33.948 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 1 -- 172.21.15.67:0/3707839411 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877705610 0 0) 0x7fc5e40afdd0 con 0x7fc5e4149dc0 2024-03-29T11:22:33.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:33.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:33.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:33.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:22:33.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 1 -- 172.21.15.67:0/3707839411 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5e40aae60 con 0x7fc5e4149dc0 2024-03-29T11:22:33.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 1 -- 172.21.15.67:0/3707839411 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964944516 0 0) 0x7fc5e40b4310 con 0x7fc5e40a9930 2024-03-29T11:22:33.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:33.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:33.950 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:33.950 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:22:33.950 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.972+0000 7fc5e1ffb640 1 -- 172.21.15.67:0/3707839411 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5e40afdd0 con 0x7fc5e40a9930 2024-03-29T11:22:33.951 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 1 -- 172.21.15.67:0/3707839411 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3798384629 0 0) 0x7fc5e40afdd0 con 0x7fc5e40a9930 2024-03-29T11:22:33.951 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 1 -- 172.21.15.67:0/3707839411 >> v1:172.21.15.67:6790/0 conn(0x7fc5e4149dc0 legacy=0x7fc5e414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.951 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 1 -- 172.21.15.67:0/3707839411 >> v1:172.21.15.67:6789/0 conn(0x7fc5e414ff40 legacy=0x7fc5e4152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.951 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:33.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 1 monclient: found mon.noname-c 2024-03-29T11:22:33.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:33.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:22:33.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 1 -- 172.21.15.67:0/3707839411 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e414a950 con 0x7fc5e40a9930 2024-03-29T11:22:33.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:33.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:33.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:33.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e950f640 5 monclient: authenticate success, global_id 4324 2024-03-29T11:22:33.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e950f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:33.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 1 -- 172.21.15.67:0/3707839411 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc5dc0018b0 con 0x7fc5e40a9930 2024-03-29T11:22:33.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:33.953 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 10 monclient: dump: 2024-03-29T11:22:33.953 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:33.953 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:33.953 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:33.953 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:33.953 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.953 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:33.953 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:33.953 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:33.953 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:33.953 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:33.953 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e950f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:33.954 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 1 -- 172.21.15.67:0/3707839411 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5dc001ad0 con 0x7fc5e40a9930 2024-03-29T11:22:33.954 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:33.954 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e950f640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:33.954 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.975+0000 7fc5e950f640 4 set_mon_vals no callback set 2024-03-29T11:22:33.954 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.976+0000 7fc5e950f640 10 monclient: shutdown 2024-03-29T11:22:33.954 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.976+0000 7fc5e950f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:33.954 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.976+0000 7fc5e950f640 1 -- 172.21.15.67:0/3707839411 >> v1:172.21.15.82:6789/0 conn(0x7fc5e40a9930 legacy=0x7fc5e4143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.956 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.977+0000 7fc5e950f640 1 -- 172.21.15.67:0/3707839411 shutdown_connections 2024-03-29T11:22:33.956 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.977+0000 7fc5e950f640 1 -- 172.21.15.67:0/3707839411 >> 172.21.15.67:0/3707839411 conn(0x7fc5e40b35e0 msgr2=0x7fc5e40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:33.956 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.979+0000 7fc5e950f640 1 -- 172.21.15.67:0/3707839411 shutdown_connections 2024-03-29T11:22:33.958 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.979+0000 7fc5e950f640 1 -- 172.21.15.67:0/3707839411 wait complete. 2024-03-29T11:22:33.958 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.980+0000 7fc5e950f640 10 monclient: build_initial_monmap 2024-03-29T11:22:33.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.980+0000 7fc5e950f640 10 monclient: monmap: 2024-03-29T11:22:33.959 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:33.959 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:33.959 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:22:33.959 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:22:33.959 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:33.959 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:33.959 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:33.959 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:22:33.959 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:22:33.959 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:33.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.980+0000 7fc5e950f640 1 librados: starting msgr at 2024-03-29T11:22:33.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.980+0000 7fc5e950f640 1 librados: starting objecter 2024-03-29T11:22:33.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.981+0000 7fc5e950f640 1 Processor -- start 2024-03-29T11:22:33.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.981+0000 7fc5e950f640 1 -- start start 2024-03-29T11:22:33.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.981+0000 7fc5e950f640 1 librados: setting wanted keys 2024-03-29T11:22:33.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.981+0000 7fc5e950f640 1 librados: calling monclient init 2024-03-29T11:22:33.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.981+0000 7fc5e950f640 10 monclient: init 2024-03-29T11:22:33.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.982+0000 7fc5e950f640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:33.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.982+0000 7fc5e950f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:33.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.982+0000 7fc5e950f640 10 monclient(hunting): picked mon.noname-c con 0x7fc5e40a9930 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:33.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.982+0000 7fc5e950f640 10 monclient(hunting): picked mon.noname-a con 0x7fc5e4149dc0 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:33.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.982+0000 7fc5e950f640 10 monclient(hunting): picked mon.noname-b con 0x7fc5e414ff40 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:33.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.982+0000 7fc5e950f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e414aae0 con 0x7fc5e4149dc0 2024-03-29T11:22:33.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.982+0000 7fc5e950f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e408be70 con 0x7fc5e414ff40 2024-03-29T11:22:33.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.982+0000 7fc5e950f640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e4147030 con 0x7fc5e40a9930 2024-03-29T11:22:33.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.982+0000 7fc5e950f640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:33.961 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.982+0000 7fc5e950f640 10 monclient(hunting): authenticate will time out at 1014.277984s 2024-03-29T11:22:33.962 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.986+0000 7fc5e27fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc5e4149dc0 0x7fc5e4147fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50439/0 (socket says 172.21.15.67:50439) 2024-03-29T11:22:33.962 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.986+0000 7fc5e27fc640 1 -- 172.21.15.67:0/1661456431 learned_addr learned my addr 172.21.15.67:0/1661456431 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:33.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.988+0000 7fc5cb7fe640 10 client.?.objecter ms_handle_connect 0x7fc5e4149dc0 2024-03-29T11:22:33.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.988+0000 7fc5cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:33.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.988+0000 7fc5cb7fe640 10 client.?.objecter ms_handle_connect 0x7fc5e40a9930 2024-03-29T11:22:33.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.988+0000 7fc5cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:33.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.988+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084922922 0 0) 0x7fc5e4147030 con 0x7fc5e40a9930 2024-03-29T11:22:33.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.988+0000 7fc5cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:33.970 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.988+0000 7fc5cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:33.970 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.988+0000 7fc5cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:33.970 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.988+0000 7fc5cb7fe640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:22:33.970 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.988+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d0004eb0 con 0x7fc5e40a9930 2024-03-29T11:22:33.970 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1207259291 0 0) 0x7fc5d0004eb0 con 0x7fc5e40a9930 2024-03-29T11:22:33.970 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5e4147030 con 0x7fc5e40a9930 2024-03-29T11:22:33.970 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc5dc0020e0 con 0x7fc5e40a9930 2024-03-29T11:22:33.970 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.970 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:33.970 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:33.971 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:33.971 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:33.971 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:33.971 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:33.971 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.971 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:33.971 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:33.971 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:33.971 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:33.971 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:33.971 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:33.972 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2988150733 0 0) 0x7fc5e4147030 con 0x7fc5e40a9930 2024-03-29T11:22:33.972 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 >> v1:172.21.15.82:6789/0 conn(0x7fc5e414ff40 legacy=0x7fc5e41486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.972 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 >> v1:172.21.15.67:6789/0 conn(0x7fc5e4149dc0 legacy=0x7fc5e4147fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.972 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:33.972 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 1 monclient: found mon.c 2024-03-29T11:22:33.972 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:33.972 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:22:33.972 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e4147310 con 0x7fc5e40a9930 2024-03-29T11:22:33.972 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:33.972 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:33.972 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:33.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5dc016050 con 0x7fc5e40a9930 2024-03-29T11:22:33.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:33.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc5dc0164a0 con 0x7fc5e40a9930 2024-03-29T11:22:33.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:33.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.989+0000 7fc5cb7fe640 10 monclient: dump: 2024-03-29T11:22:33.973 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:33.973 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:33.973 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:33.973 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:33.973 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.974 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:33.974 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:33.974 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:33.974 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:33.974 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:33.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.990+0000 7fc5e950f640 5 monclient: authenticate success, global_id 4352 2024-03-29T11:22:33.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.990+0000 7fc5e950f640 10 monclient: _renew_subs 2024-03-29T11:22:33.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.990+0000 7fc5e950f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:22:33.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.990+0000 7fc5e950f640 1 -- 172.21.15.67:0/1661456431 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e41475f0 con 0x7fc5e40a9930 2024-03-29T11:22:33.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.990+0000 7fc5e950f640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:33.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.990+0000 7fc5e950f640 10 monclient: _renew_subs 2024-03-29T11:22:33.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.990+0000 7fc5e950f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:22:33.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.990+0000 7fc5e950f640 1 -- 172.21.15.67:0/1661456431 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e408b8e0 con 0x7fc5e40a9930 2024-03-29T11:22:33.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.990+0000 7fc5e950f640 1 librados: init done 2024-03-29T11:22:33.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.991+0000 7fc5e0ff9640 4 set_mon_vals no callback set 2024-03-29T11:22:33.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.992+0000 7fc5e950f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:33.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.992+0000 7fc5e950f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:33.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.992+0000 7fc5e950f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:22:33.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.992+0000 7fc5e950f640 1 -- 172.21.15.67:0/1661456431 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c4005080 con 0x7fc5e40a9930 2024-03-29T11:22:33.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.994+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fc5dc001c40 con 0x7fc5e40a9930 2024-03-29T11:22:33.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.995+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fc5dc095f20 con 0x7fc5e40a9930 2024-03-29T11:22:33.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.995+0000 7fc5cb7fe640 10 client.4352.objecter ms_dispatch 0x7fc5e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:33.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.995+0000 7fc5cb7fe640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:33.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.995+0000 7fc5cb7fe640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:33.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.995+0000 7fc5cb7fe640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T11:22:33.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.995+0000 7fc5e27fc640 10 monclient: get_auth_request con 0x7fc5d006dc60 auth_method 0 2024-03-29T11:22:33.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:33.995+0000 7fc5cb7fe640 10 client.4352.objecter ms_handle_connect 0x7fc5d006dc60 2024-03-29T11:22:33.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.004+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5dc061800 con 0x7fc5e40a9930 2024-03-29T11:22:33.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.004+0000 7fc5cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:33.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.005+0000 7fc5cb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:34.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.340+0000 7fc5e950f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:34.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.340+0000 7fc5e950f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:34.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.340+0000 7fc5e950f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:22:34.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.340+0000 7fc5e950f640 1 -- 172.21.15.67:0/1661456431 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5c4005330 con 0x7fc5e40a9930 2024-03-29T11:22:34.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.344+0000 7fc5cb7fe640 1 -- 172.21.15.67:0/1661456431 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fc5dc063730 con 0x7fc5e40a9930 2024-03-29T11:22:34.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.344+0000 7fc5cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:34.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.344+0000 7fc5cb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:34.320 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:22:34.321 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:34.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.347+0000 7fc5e950f640 10 librados: watch_flush enter 2024-03-29T11:22:34.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.347+0000 7fc5e950f640 10 librados: watch_flush exit 2024-03-29T11:22:34.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.347+0000 7fc5e950f640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T11:22:34.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.347+0000 7fc5e950f640 10 client.4352.objecter successfully canceled tick 2024-03-29T11:22:34.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.347+0000 7fc5e950f640 1 -- 172.21.15.67:0/1661456431 >> v1:172.21.15.67:6816/404787197 conn(0x7fc5d006dc60 legacy=0x7fc5d0070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:34.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.347+0000 7fc5e950f640 10 monclient: shutdown 2024-03-29T11:22:34.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.347+0000 7fc5e950f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:34.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.347+0000 7fc5e950f640 1 -- 172.21.15.67:0/1661456431 >> v1:172.21.15.67:6790/0 conn(0x7fc5e40a9930 legacy=0x7fc5e41452e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:34.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.348+0000 7fc5e950f640 1 -- 172.21.15.67:0/1661456431 shutdown_connections 2024-03-29T11:22:34.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.348+0000 7fc5e950f640 1 -- 172.21.15.67:0/1661456431 >> 172.21.15.67:0/1661456431 conn(0x7fc5e40b35e0 msgr2=0x7fc5e40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:34.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.348+0000 7fc5e950f640 1 -- 172.21.15.67:0/1661456431 shutdown_connections 2024-03-29T11:22:34.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.348+0000 7fc5e950f640 1 -- 172.21.15.67:0/1661456431 wait complete. 2024-03-29T11:22:34.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:34.348+0000 7fc5e950f640 1 librados: shutdown 2024-03-29T11:22:34.342 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:41.149 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:41.306 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.329+0000 7ff4e8284640 10 monclient: get_monmap_and_config 2024-03-29T11:22:41.306 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.329+0000 7ff4e8284640 10 monclient: build_initial_monmap 2024-03-29T11:22:41.306 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.329+0000 7ff4e8284640 10 monclient: monmap: 2024-03-29T11:22:41.306 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:41.306 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:41.306 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:22:41.329935+0000 2024-03-29T11:22:41.306 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:22:41.329935+0000 2024-03-29T11:22:41.307 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:41.307 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:41.307 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:41.307 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:22:41.307 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:22:41.307 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:41.308 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.330+0000 7ff4e8284640 1 Processor -- start 2024-03-29T11:22:41.308 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.330+0000 7ff4e8284640 1 -- start start 2024-03-29T11:22:41.308 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.330+0000 7ff4e8284640 10 monclient: init 2024-03-29T11:22:41.308 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.331+0000 7ff4e8284640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:41.308 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.331+0000 7ff4e8284640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:41.308 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.331+0000 7ff4e8284640 10 monclient(hunting): picked mon.noname-c con 0x7ff4e00af290 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:41.308 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.331+0000 7ff4e8284640 10 monclient(hunting): picked mon.noname-a con 0x7ff4e0143ec0 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:41.311 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.335+0000 7ff4e581f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff4e0143ec0 0x7ff4e00b40d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49829/0 (socket says 172.21.15.67:49829) 2024-03-29T11:22:41.315 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.338+0000 7ff4e6020640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7ff4e00af290 0x7ff4e01437b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:48143/0 (socket says 172.21.15.67:48143) 2024-03-29T11:22:41.315 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.338+0000 7ff4e8284640 10 monclient(hunting): picked mon.noname-b con 0x7ff4e00af670 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:41.315 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.338+0000 7ff4e581f640 1 -- 172.21.15.67:0/2713080099 learned_addr learned my addr 172.21.15.67:0/2713080099 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:41.315 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.338+0000 7ff4e8284640 1 -- 172.21.15.67:0/2713080099 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e00ae740 con 0x7ff4e0143ec0 2024-03-29T11:22:41.315 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.339+0000 7ff4e8284640 1 -- 172.21.15.67:0/2713080099 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e00b06a0 con 0x7ff4e00af290 2024-03-29T11:22:41.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.339+0000 7ff4e8284640 1 -- 172.21.15.67:0/2713080099 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e00b0980 con 0x7ff4e00af670 2024-03-29T11:22:41.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.339+0000 7ff4e8284640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:41.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.339+0000 7ff4e8284640 10 monclient(hunting): authenticate will time out at 1021.634954s 2024-03-29T11:22:41.316 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 1 -- 172.21.15.67:0/2713080099 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658317406 0 0) 0x7ff4e00ae740 con 0x7ff4e0143ec0 2024-03-29T11:22:41.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 10 monclient(hunting): _negotiate 2024-03-29T11:22:41.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:41.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:41.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:22:41.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 1 -- 172.21.15.67:0/2713080099 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4cc003520 con 0x7ff4e0143ec0 2024-03-29T11:22:41.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 1 -- 172.21.15.67:0/2713080099 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1772802049 0 0) 0x7ff4cc003520 con 0x7ff4e0143ec0 2024-03-29T11:22:41.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 1 -- 172.21.15.67:0/2713080099 >> v1:172.21.15.67:6790/0 conn(0x7ff4e00af670 legacy=0x7ff4e0151dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:41.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 1 -- 172.21.15.67:0/2713080099 >> v1:172.21.15.82:6789/0 conn(0x7ff4e00af290 legacy=0x7ff4e01437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 10 monclient: _finish_hunting 0 2024-03-29T11:22:41.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 1 monclient: found mon.noname-a 2024-03-29T11:22:41.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:41.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:22:41.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 1 -- 172.21.15.67:0/2713080099 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e00b0c60 con 0x7ff4e0143ec0 2024-03-29T11:22:41.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 10 monclient: _finish_auth 0 2024-03-29T11:22:41.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 10 monclient: _check_auth_tickets 2024-03-29T11:22:41.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:41.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 1 -- 172.21.15.67:0/2713080099 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff4dc001690 con 0x7ff4e0143ec0 2024-03-29T11:22:41.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:41.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:41.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.340+0000 7ff4e501e640 10 monclient: dump: 2024-03-29T11:22:41.319 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:41.320 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:41.320 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:41.320 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:41.320 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:41.320 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:41.320 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:41.320 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:41.320 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:41.320 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:41.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.341+0000 7ff4e501e640 1 -- 172.21.15.67:0/2713080099 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4dc001920 con 0x7ff4e0143ec0 2024-03-29T11:22:41.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.341+0000 7ff4e501e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:41.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.341+0000 7ff4e501e640 1 -- 172.21.15.67:0/2713080099 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff4dc001cb0 con 0x7ff4e0143ec0 2024-03-29T11:22:41.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.341+0000 7ff4e501e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:41.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.341+0000 7ff4e501e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:41.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.341+0000 7ff4e501e640 10 monclient: dump: 2024-03-29T11:22:41.321 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:41.321 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:41.321 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:41.321 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:41.321 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:41.321 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:41.321 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:41.321 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:41.321 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:41.322 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:41.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.341+0000 7ff4e8284640 5 monclient: authenticate success, global_id 4344 2024-03-29T11:22:41.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.341+0000 7ff4e8284640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:41.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.341+0000 7ff4e8284640 4 set_mon_vals no callback set 2024-03-29T11:22:41.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.341+0000 7ff4e8284640 10 monclient: shutdown 2024-03-29T11:22:41.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.341+0000 7ff4e8284640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:41.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.341+0000 7ff4e8284640 1 -- 172.21.15.67:0/2713080099 >> v1:172.21.15.67:6789/0 conn(0x7ff4e0143ec0 legacy=0x7ff4e00b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.344+0000 7ff4e8284640 1 -- 172.21.15.67:0/2713080099 shutdown_connections 2024-03-29T11:22:41.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.344+0000 7ff4e8284640 1 -- 172.21.15.67:0/2713080099 >> 172.21.15.67:0/2713080099 conn(0x7ff4e0066a00 msgr2=0x7ff4e0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:41.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.344+0000 7ff4e8284640 1 -- 172.21.15.67:0/2713080099 shutdown_connections 2024-03-29T11:22:41.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.344+0000 7ff4e8284640 1 -- 172.21.15.67:0/2713080099 wait complete. 2024-03-29T11:22:41.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.345+0000 7ff4e8284640 10 monclient: build_initial_monmap 2024-03-29T11:22:41.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.345+0000 7ff4e8284640 10 monclient: monmap: 2024-03-29T11:22:41.323 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:41.323 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:41.323 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:22:41.324 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:22:41.324 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:41.324 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:41.324 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:41.324 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:22:41.324 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:22:41.324 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:41.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.345+0000 7ff4e8284640 1 librados: starting msgr at 2024-03-29T11:22:41.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.345+0000 7ff4e8284640 1 librados: starting objecter 2024-03-29T11:22:41.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.345+0000 7ff4e8284640 1 Processor -- start 2024-03-29T11:22:41.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.345+0000 7ff4e8284640 1 -- start start 2024-03-29T11:22:41.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.345+0000 7ff4e8284640 1 librados: setting wanted keys 2024-03-29T11:22:41.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.345+0000 7ff4e8284640 1 librados: calling monclient init 2024-03-29T11:22:41.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.345+0000 7ff4e8284640 10 monclient: init 2024-03-29T11:22:41.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.346+0000 7ff4e8284640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:41.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.346+0000 7ff4e8284640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:41.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.346+0000 7ff4e8284640 10 monclient(hunting): picked mon.noname-c con 0x7ff4e00af290 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:41.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.347+0000 7ff4e8284640 10 monclient(hunting): picked mon.noname-b con 0x7ff4e00af670 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:41.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.347+0000 7ff4e8284640 10 monclient(hunting): picked mon.noname-a con 0x7ff4e0143ec0 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:41.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.347+0000 7ff4e8284640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e0146f10 con 0x7ff4e0143ec0 2024-03-29T11:22:41.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.347+0000 7ff4e8284640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e013f900 con 0x7ff4e00af670 2024-03-29T11:22:41.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.347+0000 7ff4e8284640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e013fa90 con 0x7ff4e00af290 2024-03-29T11:22:41.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.347+0000 7ff4e8284640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:41.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.347+0000 7ff4e8284640 10 monclient(hunting): authenticate will time out at 1021.643521s 2024-03-29T11:22:41.328 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.351+0000 7ff4e581f640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7ff4e00af670 0x7ff4e01393e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:48211/0 (socket says 172.21.15.67:48211) 2024-03-29T11:22:41.328 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.351+0000 7ff4e581f640 1 -- 172.21.15.67:0/3939810016 learned_addr learned my addr 172.21.15.67:0/3939810016 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:41.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.352+0000 7ff4d6ffd640 10 client.?.objecter ms_handle_connect 0x7ff4e00af670 2024-03-29T11:22:41.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.352+0000 7ff4d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:41.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.352+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250094319 0 0) 0x7ff4e013f900 con 0x7ff4e00af670 2024-03-29T11:22:41.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.352+0000 7ff4d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:41.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.352+0000 7ff4d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:41.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.352+0000 7ff4d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:41.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.352+0000 7ff4d6ffd640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:22:41.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.352+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c4004eb0 con 0x7ff4e00af670 2024-03-29T11:22:41.334 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 10 client.?.objecter ms_handle_connect 0x7ff4e00af290 2024-03-29T11:22:41.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:41.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501722786 0 0) 0x7ff4e013fa90 con 0x7ff4e00af290 2024-03-29T11:22:41.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:41.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:41.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:41.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:22:41.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4e013f900 con 0x7ff4e00af290 2024-03-29T11:22:41.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 397819383 0 0) 0x7ff4c4004eb0 con 0x7ff4e00af670 2024-03-29T11:22:41.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4e013fa90 con 0x7ff4e00af670 2024-03-29T11:22:41.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff4dc001550 con 0x7ff4e00af670 2024-03-29T11:22:41.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:41.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:41.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 10 monclient(hunting): dump: 2024-03-29T11:22:41.336 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:41.336 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:41.336 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:41.336 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:41.336 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:41.336 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:41.336 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:41.336 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:41.336 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:41.337 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:41.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:41.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3389378124 0 0) 0x7ff4e013f900 con 0x7ff4e00af290 2024-03-29T11:22:41.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4c4004eb0 con 0x7ff4e00af290 2024-03-29T11:22:41.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4192827561 0 0) 0x7ff4e013fa90 con 0x7ff4e00af670 2024-03-29T11:22:41.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 >> v1:172.21.15.67:6790/0 conn(0x7ff4e00af290 legacy=0x7ff4e0146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.354+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 >> v1:172.21.15.67:6789/0 conn(0x7ff4e0143ec0 legacy=0x7ff4e01db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.357+0000 7ff4d6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:41.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.357+0000 7ff4d6ffd640 1 monclient: found mon.b 2024-03-29T11:22:41.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.357+0000 7ff4d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:41.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.357+0000 7ff4d6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:41.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.357+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e01460f0 con 0x7ff4e00af670 2024-03-29T11:22:41.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.358+0000 7ff4d6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:41.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.358+0000 7ff4d6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:41.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.358+0000 7ff4d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:41.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.358+0000 7ff4e8284640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:22:41.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.358+0000 7ff4e8284640 10 monclient: _renew_subs 2024-03-29T11:22:41.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.358+0000 7ff4e8284640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:41.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.358+0000 7ff4e8284640 1 -- 172.21.15.67:0/3939810016 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e0146340 con 0x7ff4e00af670 2024-03-29T11:22:41.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.358+0000 7ff4e8284640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:41.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.358+0000 7ff4e8284640 10 monclient: _renew_subs 2024-03-29T11:22:41.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.358+0000 7ff4e8284640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:41.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.358+0000 7ff4e8284640 1 -- 172.21.15.67:0/3939810016 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e0139cd0 con 0x7ff4e00af670 2024-03-29T11:22:41.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.358+0000 7ff4e8284640 1 librados: init done 2024-03-29T11:22:41.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.361+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4dc004a90 con 0x7ff4e00af670 2024-03-29T11:22:41.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.361+0000 7ff4d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:41.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.361+0000 7ff4e481d640 4 set_mon_vals no callback set 2024-03-29T11:22:41.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.362+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff4dc004da0 con 0x7ff4e00af670 2024-03-29T11:22:41.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.362+0000 7ff4d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:41.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.362+0000 7ff4d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:41.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.362+0000 7ff4d6ffd640 10 monclient: dump: 2024-03-29T11:22:41.341 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:41.341 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:41.342 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:41.342 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:41.342 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:41.342 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:41.342 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:41.342 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:41.342 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:41.342 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:41.342 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.362+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7ff4dc01ad70 con 0x7ff4e00af670 2024-03-29T11:22:41.342 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.363+0000 7ff4e8284640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:41.342 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.363+0000 7ff4e8284640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:41.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.363+0000 7ff4e8284640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:41.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.363+0000 7ff4e8284640 1 -- 172.21.15.67:0/3939810016 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c8005080 con 0x7ff4e00af670 2024-03-29T11:22:41.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.363+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7ff4dc084950 con 0x7ff4e00af670 2024-03-29T11:22:41.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.363+0000 7ff4d6ffd640 10 client.4330.objecter ms_dispatch 0x7ff4e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:41.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.363+0000 7ff4d6ffd640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:41.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.363+0000 7ff4d6ffd640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:41.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.363+0000 7ff4d6ffd640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-29T11:22:41.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.363+0000 7ff4e6020640 10 monclient: get_auth_request con 0x7ff4c406dda0 auth_method 0 2024-03-29T11:22:41.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.364+0000 7ff4d6ffd640 10 client.4330.objecter ms_handle_connect 0x7ff4c406dda0 2024-03-29T11:22:41.347 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.371+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4dc0502f0 con 0x7ff4e00af670 2024-03-29T11:22:41.347 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.371+0000 7ff4d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:41.347 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.371+0000 7ff4d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:41.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.711+0000 7ff4e8284640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:41.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.711+0000 7ff4e8284640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:41.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.711+0000 7ff4e8284640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:22:41.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.711+0000 7ff4e8284640 1 -- 172.21.15.67:0/3939810016 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4c8005330 con 0x7ff4e00af670 2024-03-29T11:22:41.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.712+0000 7ff4d6ffd640 1 -- 172.21.15.67:0/3939810016 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7ff4dc052220 con 0x7ff4e00af670 2024-03-29T11:22:41.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.712+0000 7ff4d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:41.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.712+0000 7ff4d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:41.690 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:22:41.690 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:41.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.716+0000 7ff4e8284640 10 librados: watch_flush enter 2024-03-29T11:22:41.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.716+0000 7ff4e8284640 10 librados: watch_flush exit 2024-03-29T11:22:41.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.716+0000 7ff4e8284640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-29T11:22:41.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.716+0000 7ff4e8284640 10 client.4330.objecter successfully canceled tick 2024-03-29T11:22:41.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.716+0000 7ff4e8284640 1 -- 172.21.15.67:0/3939810016 >> v1:172.21.15.67:6816/404787197 conn(0x7ff4c406dda0 legacy=0x7ff4c4070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.717+0000 7ff4e8284640 10 monclient: shutdown 2024-03-29T11:22:41.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.717+0000 7ff4e8284640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:41.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.717+0000 7ff4e8284640 1 -- 172.21.15.67:0/3939810016 >> v1:172.21.15.82:6789/0 conn(0x7ff4e00af670 legacy=0x7ff4e01393e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.717+0000 7ff4e8284640 1 -- 172.21.15.67:0/3939810016 shutdown_connections 2024-03-29T11:22:41.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.717+0000 7ff4e8284640 1 -- 172.21.15.67:0/3939810016 >> 172.21.15.67:0/3939810016 conn(0x7ff4e0066a00 msgr2=0x7ff4e00b3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:41.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.717+0000 7ff4e8284640 1 -- 172.21.15.67:0/3939810016 shutdown_connections 2024-03-29T11:22:41.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.717+0000 7ff4e8284640 1 -- 172.21.15.67:0/3939810016 wait complete. 2024-03-29T11:22:41.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:41.717+0000 7ff4e8284640 1 librados: shutdown 2024-03-29T11:22:41.708 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:48.515 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:48.666 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.688+0000 7fe739ec1640 10 monclient: get_monmap_and_config 2024-03-29T11:22:48.666 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.688+0000 7fe739ec1640 10 monclient: build_initial_monmap 2024-03-29T11:22:48.666 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.688+0000 7fe739ec1640 10 monclient: monmap: 2024-03-29T11:22:48.666 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:48.666 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:48.666 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:22:48.689598+0000 2024-03-29T11:22:48.666 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:22:48.689598+0000 2024-03-29T11:22:48.666 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:48.667 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:48.667 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:48.667 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:22:48.667 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:22:48.667 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:48.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.689+0000 7fe739ec1640 1 Processor -- start 2024-03-29T11:22:48.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.690+0000 7fe739ec1640 1 -- start start 2024-03-29T11:22:48.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.690+0000 7fe739ec1640 10 monclient: init 2024-03-29T11:22:48.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.690+0000 7fe739ec1640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:48.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.690+0000 7fe739ec1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:48.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.690+0000 7fe739ec1640 10 monclient(hunting): picked mon.noname-a con 0x7fe7340a9930 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:48.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.690+0000 7fe739ec1640 10 monclient(hunting): picked mon.noname-b con 0x7fe734149dc0 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:48.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.691+0000 7fe739ec1640 10 monclient(hunting): picked mon.noname-c con 0x7fe73414ff40 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:48.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.691+0000 7fe739ec1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7340aae60 con 0x7fe7340a9930 2024-03-29T11:22:48.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.691+0000 7fe739ec1640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7340b4310 con 0x7fe73414ff40 2024-03-29T11:22:48.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.691+0000 7fe739ec1640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7340afdd0 con 0x7fe734149dc0 2024-03-29T11:22:48.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.691+0000 7fe739ec1640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:48.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.691+0000 7fe739ec1640 10 monclient(hunting): authenticate will time out at 1028.986972s 2024-03-29T11:22:48.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.691+0000 7fe7337fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe7340a9930 0x7fe734143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38206/0 (socket says 172.21.15.67:38206) 2024-03-29T11:22:48.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.691+0000 7fe7337fe640 1 -- 172.21.15.67:0/68126547 learned_addr learned my addr 172.21.15.67:0/68126547 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:48.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.691+0000 7fe7327fc640 1 -- 172.21.15.67:0/68126547 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540848603 0 0) 0x7fe7340aae60 con 0x7fe7340a9930 2024-03-29T11:22:48.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.691+0000 7fe7327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:48.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.691+0000 7fe7327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:48.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.691+0000 7fe7327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:48.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.691+0000 7fe7327fc640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:22:48.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.691+0000 7fe7327fc640 1 -- 172.21.15.67:0/68126547 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c003520 con 0x7fe7340a9930 2024-03-29T11:22:48.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.692+0000 7fe7327fc640 1 -- 172.21.15.67:0/68126547 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1288071847 0 0) 0x7fe71c003520 con 0x7fe7340a9930 2024-03-29T11:22:48.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.692+0000 7fe7327fc640 1 -- 172.21.15.67:0/68126547 >> v1:172.21.15.67:6790/0 conn(0x7fe734149dc0 legacy=0x7fe73414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.692+0000 7fe7327fc640 1 -- 172.21.15.67:0/68126547 >> v1:172.21.15.82:6789/0 conn(0x7fe73414ff40 legacy=0x7fe734152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:48.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.694+0000 7fe7327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:48.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.694+0000 7fe7327fc640 1 monclient: found mon.noname-a 2024-03-29T11:22:48.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.694+0000 7fe7327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:48.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.694+0000 7fe7327fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:22:48.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.694+0000 7fe7327fc640 1 -- 172.21.15.67:0/68126547 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73414a950 con 0x7fe7340a9930 2024-03-29T11:22:48.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.694+0000 7fe7327fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:48.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.694+0000 7fe7327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:48.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.694+0000 7fe7327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:48.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.694+0000 7fe7327fc640 1 -- 172.21.15.67:0/68126547 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe72c001a10 con 0x7fe7340a9930 2024-03-29T11:22:48.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.694+0000 7fe7327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.694+0000 7fe7327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:48.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.694+0000 7fe7327fc640 10 monclient: dump: 2024-03-29T11:22:48.674 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:48.674 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:48.674 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:48.674 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:48.674 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.674 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:48.675 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:48.675 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:48.675 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:48.675 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:48.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.694+0000 7fe739ec1640 5 monclient: authenticate success, global_id 4347 2024-03-29T11:22:48.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.694+0000 7fe739ec1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:48.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.695+0000 7fe7327fc640 1 -- 172.21.15.67:0/68126547 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe72c001be0 con 0x7fe7340a9930 2024-03-29T11:22:48.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.695+0000 7fe7327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:48.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.695+0000 7fe7327fc640 1 -- 172.21.15.67:0/68126547 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe72c001ff0 con 0x7fe7340a9930 2024-03-29T11:22:48.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.695+0000 7fe7327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.695+0000 7fe7327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:48.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.695+0000 7fe7327fc640 10 monclient: dump: 2024-03-29T11:22:48.675 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:48.675 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:48.676 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:48.676 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:48.676 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.676 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:48.676 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:48.676 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:48.676 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:48.676 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:48.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.695+0000 7fe739ec1640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:48.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.695+0000 7fe739ec1640 4 set_mon_vals no callback set 2024-03-29T11:22:48.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.696+0000 7fe739ec1640 10 monclient: shutdown 2024-03-29T11:22:48.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.696+0000 7fe739ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:48.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.696+0000 7fe739ec1640 1 -- 172.21.15.67:0/68126547 >> v1:172.21.15.67:6789/0 conn(0x7fe7340a9930 legacy=0x7fe734143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.696+0000 7fe739ec1640 1 -- 172.21.15.67:0/68126547 shutdown_connections 2024-03-29T11:22:48.677 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.696+0000 7fe739ec1640 1 -- 172.21.15.67:0/68126547 >> 172.21.15.67:0/68126547 conn(0x7fe7340b35e0 msgr2=0x7fe7340b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:48.677 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.696+0000 7fe739ec1640 1 -- 172.21.15.67:0/68126547 shutdown_connections 2024-03-29T11:22:48.677 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.699+0000 7fe739ec1640 1 -- 172.21.15.67:0/68126547 wait complete. 2024-03-29T11:22:48.677 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.700+0000 7fe739ec1640 10 monclient: build_initial_monmap 2024-03-29T11:22:48.677 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.700+0000 7fe739ec1640 10 monclient: monmap: 2024-03-29T11:22:48.677 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:48.678 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:48.678 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:22:48.678 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:22:48.678 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:48.678 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:48.678 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:48.678 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:22:48.678 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:22:48.678 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:48.678 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.700+0000 7fe739ec1640 1 librados: starting msgr at 2024-03-29T11:22:48.678 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.700+0000 7fe739ec1640 1 librados: starting objecter 2024-03-29T11:22:48.678 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.700+0000 7fe739ec1640 1 Processor -- start 2024-03-29T11:22:48.678 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.700+0000 7fe739ec1640 1 -- start start 2024-03-29T11:22:48.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.701+0000 7fe739ec1640 1 librados: setting wanted keys 2024-03-29T11:22:48.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.701+0000 7fe739ec1640 1 librados: calling monclient init 2024-03-29T11:22:48.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.701+0000 7fe739ec1640 10 monclient: init 2024-03-29T11:22:48.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.701+0000 7fe739ec1640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:48.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.701+0000 7fe739ec1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:48.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.702+0000 7fe739ec1640 10 monclient(hunting): picked mon.noname-c con 0x7fe7340a9930 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:48.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.702+0000 7fe739ec1640 10 monclient(hunting): picked mon.noname-a con 0x7fe734149dc0 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:48.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.702+0000 7fe739ec1640 10 monclient(hunting): picked mon.noname-b con 0x7fe73414ff40 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:48.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.702+0000 7fe739ec1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7341db620 con 0x7fe734149dc0 2024-03-29T11:22:48.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.702+0000 7fe739ec1640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe734144510 con 0x7fe73414ff40 2024-03-29T11:22:48.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.702+0000 7fe739ec1640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7340ab1d0 con 0x7fe7340a9930 2024-03-29T11:22:48.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.702+0000 7fe739ec1640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:48.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.702+0000 7fe739ec1640 10 monclient(hunting): authenticate will time out at 1028.998705s 2024-03-29T11:22:48.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.702+0000 7fe7337fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fe7340a9930 0x7fe734148bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:43434/0 (socket says 172.21.15.67:43434) 2024-03-29T11:22:48.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.702+0000 7fe7337fe640 1 -- 172.21.15.67:0/3938982380 learned_addr learned my addr 172.21.15.67:0/3938982380 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:48.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.703+0000 7fe71bfff640 10 client.?.objecter ms_handle_connect 0x7fe734149dc0 2024-03-29T11:22:48.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.703+0000 7fe71bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:48.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 10 client.?.objecter ms_handle_connect 0x7fe7340a9930 2024-03-29T11:22:48.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:48.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533527791 0 0) 0x7fe7340ab1d0 con 0x7fe7340a9930 2024-03-29T11:22:48.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:48.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:48.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:48.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:22:48.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe720004eb0 con 0x7fe7340a9930 2024-03-29T11:22:48.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451356534 0 0) 0x7fe7341db620 con 0x7fe734149dc0 2024-03-29T11:22:48.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:48.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:48.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:48.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:22:48.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7340ab1d0 con 0x7fe734149dc0 2024-03-29T11:22:48.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 10 client.?.objecter ms_handle_connect 0x7fe73414ff40 2024-03-29T11:22:48.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.704+0000 7fe71bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:48.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811745474 0 0) 0x7fe734144510 con 0x7fe73414ff40 2024-03-29T11:22:48.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:48.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:48.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:48.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:22:48.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7341db620 con 0x7fe73414ff40 2024-03-29T11:22:48.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2373102704 0 0) 0x7fe720004eb0 con 0x7fe7340a9930 2024-03-29T11:22:48.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe734144510 con 0x7fe7340a9930 2024-03-29T11:22:48.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe72c001ef0 con 0x7fe7340a9930 2024-03-29T11:22:48.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:48.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient(hunting): dump: 2024-03-29T11:22:48.690 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:48.691 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:48.691 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:48.691 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:48.691 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.691 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:48.691 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:48.691 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:48.691 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:48.691 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:48.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:48.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2814482160 0 0) 0x7fe7340ab1d0 con 0x7fe734149dc0 2024-03-29T11:22:48.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe720004eb0 con 0x7fe734149dc0 2024-03-29T11:22:48.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe728001820 con 0x7fe734149dc0 2024-03-29T11:22:48.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient(hunting): dump: 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2087540205 0 0) 0x7fe7341db620 con 0x7fe73414ff40 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7340ab1d0 con 0x7fe73414ff40 2024-03-29T11:22:48.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe724001650 con 0x7fe73414ff40 2024-03-29T11:22:48.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:48.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient(hunting): dump: 2024-03-29T11:22:48.693 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:48.693 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:48.693 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:48.693 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:48.693 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.693 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:48.693 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:48.693 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:48.693 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:48.693 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:48.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2745544511 0 0) 0x7fe720004eb0 con 0x7fe734149dc0 2024-03-29T11:22:48.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 >> v1:172.21.15.67:6790/0 conn(0x7fe7340a9930 legacy=0x7fe734148bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 >> v1:172.21.15.82:6789/0 conn(0x7fe73414ff40 legacy=0x7fe73413a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:48.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 monclient: found mon.a 2024-03-29T11:22:48.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:48.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:48.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7341492e0 con 0x7fe734149dc0 2024-03-29T11:22:48.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient: _finish_auth 0 2024-03-29T11:22:48.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:48.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe71bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:48.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe739ec1640 5 monclient: authenticate success, global_id 4350 2024-03-29T11:22:48.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe739ec1640 10 monclient: _renew_subs 2024-03-29T11:22:48.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe739ec1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:48.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.705+0000 7fe739ec1640 1 -- 172.21.15.67:0/3938982380 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7341dc1c0 con 0x7fe734149dc0 2024-03-29T11:22:48.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.706+0000 7fe739ec1640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:48.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.706+0000 7fe739ec1640 10 monclient: _renew_subs 2024-03-29T11:22:48.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.706+0000 7fe739ec1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:48.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.706+0000 7fe739ec1640 1 -- 172.21.15.67:0/3938982380 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe734148680 con 0x7fe734149dc0 2024-03-29T11:22:48.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.706+0000 7fe739ec1640 1 librados: init done 2024-03-29T11:22:48.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.707+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe728001980 con 0x7fe734149dc0 2024-03-29T11:22:48.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.707+0000 7fe71bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:48.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.707+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe728003810 con 0x7fe734149dc0 2024-03-29T11:22:48.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.707+0000 7fe71bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.707+0000 7fe71bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:48.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.707+0000 7fe71bfff640 10 monclient: dump: 2024-03-29T11:22:48.695 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:48.695 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.708+0000 7fe739ec1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.708+0000 7fe739ec1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.708+0000 7fe739ec1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.708+0000 7fe739ec1640 1 -- 172.21.15.67:0/3938982380 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe734144f40 con 0x7fe734149dc0 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.708+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fe728019820 con 0x7fe734149dc0 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.709+0000 7fe731ffb640 4 set_mon_vals no callback set 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.709+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fe7280837e0 con 0x7fe734149dc0 2024-03-29T11:22:48.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.709+0000 7fe71bfff640 10 client.4350.objecter ms_dispatch 0x7fe734000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:48.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.709+0000 7fe71bfff640 3 client.4350.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:48.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.709+0000 7fe71bfff640 3 client.4350.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:48.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.709+0000 7fe71bfff640 20 client.4350.objecter dump_active .. 0 homeless 2024-03-29T11:22:48.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.710+0000 7fe7337fe640 10 monclient: get_auth_request con 0x7fe72006e0b0 auth_method 0 2024-03-29T11:22:48.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.710+0000 7fe71bfff640 10 client.4350.objecter ms_handle_connect 0x7fe72006e0b0 2024-03-29T11:22:48.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.717+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe72804eef0 con 0x7fe734149dc0 2024-03-29T11:22:48.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.717+0000 7fe71bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:48.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:48.717+0000 7fe71bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:49.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.060+0000 7fe739ec1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:49.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.060+0000 7fe739ec1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:49.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.060+0000 7fe739ec1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:49.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.060+0000 7fe739ec1640 1 -- 172.21.15.67:0/3938982380 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7341db7b0 con 0x7fe734149dc0 2024-03-29T11:22:49.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.062+0000 7fe71bfff640 1 -- 172.21.15.67:0/3938982380 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fe728050ff0 con 0x7fe734149dc0 2024-03-29T11:22:49.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.062+0000 7fe71bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:49.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.062+0000 7fe71bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:49.039 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:22:49.040 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:49.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.065+0000 7fe739ec1640 10 librados: watch_flush enter 2024-03-29T11:22:49.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.065+0000 7fe739ec1640 10 librados: watch_flush exit 2024-03-29T11:22:49.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.065+0000 7fe739ec1640 20 client.4350.objecter shutdown clearing up homeless session... 2024-03-29T11:22:49.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.065+0000 7fe739ec1640 10 client.4350.objecter successfully canceled tick 2024-03-29T11:22:49.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.065+0000 7fe739ec1640 1 -- 172.21.15.67:0/3938982380 >> v1:172.21.15.67:6816/404787197 conn(0x7fe72006e0b0 legacy=0x7fe720070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:49.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.065+0000 7fe739ec1640 10 monclient: shutdown 2024-03-29T11:22:49.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.065+0000 7fe739ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:49.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.065+0000 7fe739ec1640 1 -- 172.21.15.67:0/3938982380 >> v1:172.21.15.67:6789/0 conn(0x7fe734149dc0 legacy=0x7fe7341dbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:49.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.066+0000 7fe739ec1640 1 -- 172.21.15.67:0/3938982380 shutdown_connections 2024-03-29T11:22:49.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.066+0000 7fe739ec1640 1 -- 172.21.15.67:0/3938982380 >> 172.21.15.67:0/3938982380 conn(0x7fe7340b35e0 msgr2=0x7fe734140a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:49.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.066+0000 7fe739ec1640 1 -- 172.21.15.67:0/3938982380 shutdown_connections 2024-03-29T11:22:49.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.070+0000 7fe739ec1640 1 -- 172.21.15.67:0/3938982380 wait complete. 2024-03-29T11:22:49.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:49.070+0000 7fe739ec1640 1 librados: shutdown 2024-03-29T11:22:49.070 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:55.877 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:56.024 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.047+0000 7fe01df93640 10 monclient: get_monmap_and_config 2024-03-29T11:22:56.024 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.047+0000 7fe01df93640 10 monclient: build_initial_monmap 2024-03-29T11:22:56.024 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.047+0000 7fe01df93640 10 monclient: monmap: 2024-03-29T11:22:56.024 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:56.024 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:56.025 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:22:56.048043+0000 2024-03-29T11:22:56.025 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:22:56.048043+0000 2024-03-29T11:22:56.025 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:56.025 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:56.025 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:56.025 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:22:56.025 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:22:56.025 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:56.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.048+0000 7fe01df93640 1 Processor -- start 2024-03-29T11:22:56.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.048+0000 7fe01df93640 1 -- start start 2024-03-29T11:22:56.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.048+0000 7fe01df93640 10 monclient: init 2024-03-29T11:22:56.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.049+0000 7fe01df93640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:56.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.049+0000 7fe01df93640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:56.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.049+0000 7fe01df93640 10 monclient(hunting): picked mon.noname-b con 0x7fe0180b0770 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:56.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.049+0000 7fe01df93640 10 monclient(hunting): picked mon.noname-a con 0x7fe01814a700 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:56.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.049+0000 7fe01df93640 10 monclient(hunting): picked mon.noname-c con 0x7fe0180ac5a0 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:56.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.049+0000 7fe01df93640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0180aba70 con 0x7fe01814a700 2024-03-29T11:22:56.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.049+0000 7fe01df93640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0180aecb0 con 0x7fe0180ac5a0 2024-03-29T11:22:56.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.049+0000 7fe01df93640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0180af150 con 0x7fe0180b0770 2024-03-29T11:22:56.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.049+0000 7fe01df93640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:56.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.049+0000 7fe01df93640 10 monclient(hunting): authenticate will time out at 1036.345336s 2024-03-29T11:22:56.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.049+0000 7fe017fff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fe0180ac5a0 0x7fe018151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:45046/0 (socket says 172.21.15.67:45046) 2024-03-29T11:22:56.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.049+0000 7fe017fff640 1 -- 172.21.15.67:0/119268282 learned_addr learned my addr 172.21.15.67:0/119268282 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:56.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.050+0000 7fe0167fc640 1 -- 172.21.15.67:0/119268282 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035653681 0 0) 0x7fe0180aba70 con 0x7fe01814a700 2024-03-29T11:22:56.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.050+0000 7fe0167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:56.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.050+0000 7fe0167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:56.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.050+0000 7fe0167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:56.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.050+0000 7fe0167fc640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:22:56.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.050+0000 7fe0167fc640 1 -- 172.21.15.67:0/119268282 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe000003520 con 0x7fe01814a700 2024-03-29T11:22:56.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.050+0000 7fe0167fc640 1 -- 172.21.15.67:0/119268282 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461645786 0 0) 0x7fe0180af150 con 0x7fe0180b0770 2024-03-29T11:22:56.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.050+0000 7fe0167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:56.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.050+0000 7fe0167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:56.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.050+0000 7fe0167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:56.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.050+0000 7fe0167fc640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:22:56.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.050+0000 7fe0167fc640 1 -- 172.21.15.67:0/119268282 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0180aba70 con 0x7fe0180b0770 2024-03-29T11:22:56.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 1 -- 172.21.15.67:0/119268282 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4097931232 0 0) 0x7fe000003520 con 0x7fe01814a700 2024-03-29T11:22:56.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 1 -- 172.21.15.67:0/119268282 >> v1:172.21.15.67:6790/0 conn(0x7fe0180b0770 legacy=0x7fe018149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 1 -- 172.21.15.67:0/119268282 >> v1:172.21.15.82:6789/0 conn(0x7fe0180ac5a0 legacy=0x7fe018151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:56.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 1 monclient: found mon.noname-a 2024-03-29T11:22:56.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:56.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:22:56.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 1 -- 172.21.15.67:0/119268282 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0180ac9d0 con 0x7fe01814a700 2024-03-29T11:22:56.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:56.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:56.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:56.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 1 -- 172.21.15.67:0/119268282 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe00c001690 con 0x7fe01814a700 2024-03-29T11:22:56.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:56.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe0167fc640 10 monclient: dump: 2024-03-29T11:22:56.038 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:56.038 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:56.038 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:56.038 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:56.038 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.038 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:56.038 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:56.038 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:56.038 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:56.038 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:56.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe01df93640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:22:56.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.051+0000 7fe01df93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:56.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.052+0000 7fe0167fc640 1 -- 172.21.15.67:0/119268282 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe00c0018b0 con 0x7fe01814a700 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.052+0000 7fe0167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.052+0000 7fe0167fc640 1 -- 172.21.15.67:0/119268282 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe00c001c40 con 0x7fe01814a700 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.052+0000 7fe0167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.052+0000 7fe0167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.052+0000 7fe0167fc640 10 monclient: dump: 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:56.039 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.052+0000 7fe01df93640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.052+0000 7fe01df93640 4 set_mon_vals no callback set 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.053+0000 7fe01df93640 10 monclient: shutdown 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.053+0000 7fe01df93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.053+0000 7fe01df93640 1 -- 172.21.15.67:0/119268282 >> v1:172.21.15.67:6789/0 conn(0x7fe01814a700 legacy=0x7fe0180abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.053+0000 7fe01df93640 1 -- 172.21.15.67:0/119268282 shutdown_connections 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.053+0000 7fe01df93640 1 -- 172.21.15.67:0/119268282 >> 172.21.15.67:0/119268282 conn(0x7fe01806b660 msgr2=0x7fe01806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.053+0000 7fe01df93640 1 -- 172.21.15.67:0/119268282 shutdown_connections 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.053+0000 7fe01df93640 1 -- 172.21.15.67:0/119268282 wait complete. 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.054+0000 7fe01df93640 10 monclient: build_initial_monmap 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.054+0000 7fe01df93640 10 monclient: monmap: 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:22:56.040 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.054+0000 7fe01df93640 1 librados: starting msgr at 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.054+0000 7fe01df93640 1 librados: starting objecter 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.054+0000 7fe01df93640 1 Processor -- start 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.055+0000 7fe01df93640 1 -- start start 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.055+0000 7fe01df93640 1 librados: setting wanted keys 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.055+0000 7fe01df93640 1 librados: calling monclient init 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.055+0000 7fe01df93640 10 monclient: init 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.055+0000 7fe01df93640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.055+0000 7fe01df93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:56.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.056+0000 7fe01df93640 10 monclient(hunting): picked mon.noname-a con 0x7fe0180ac5a0 addr v1:172.21.15.67:6789/0 2024-03-29T11:22:56.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.056+0000 7fe01df93640 10 monclient(hunting): picked mon.noname-b con 0x7fe0180b0770 addr v1:172.21.15.82:6789/0 2024-03-29T11:22:56.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.056+0000 7fe0177fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe0180ac5a0 0x7fe0181264b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34434/0 (socket says 172.21.15.67:34434) 2024-03-29T11:22:56.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.056+0000 7fe0177fe640 1 -- 172.21.15.67:0/2345492090 learned_addr learned my addr 172.21.15.67:0/2345492090 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:22:56.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.056+0000 7fe01df93640 10 monclient(hunting): picked mon.noname-c con 0x7fe01814a700 addr v1:172.21.15.67:6790/0 2024-03-29T11:22:56.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.056+0000 7fe01df93640 1 -- 172.21.15.67:0/2345492090 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe018125f90 con 0x7fe0180ac5a0 2024-03-29T11:22:56.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.056+0000 7fe01df93640 1 -- 172.21.15.67:0/2345492090 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe018125b40 con 0x7fe0180b0770 2024-03-29T11:22:56.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.056+0000 7fdffffff640 10 client.?.objecter ms_handle_connect 0x7fe0180ac5a0 2024-03-29T11:22:56.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.056+0000 7fdffffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:56.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.056+0000 7fdffffff640 10 client.?.objecter ms_handle_connect 0x7fe01814a700 2024-03-29T11:22:56.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.056+0000 7fdffffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:56.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.056+0000 7fe01df93640 1 -- 172.21.15.67:0/2345492090 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe018125cd0 con 0x7fe01814a700 2024-03-29T11:22:56.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.056+0000 7fe01df93640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:56.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.056+0000 7fe01df93640 10 monclient(hunting): authenticate will time out at 1036.352876s 2024-03-29T11:22:56.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220630710 0 0) 0x7fe018125f90 con 0x7fe0180ac5a0 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe004004eb0 con 0x7fe0180ac5a0 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862362977 0 0) 0x7fe018125cd0 con 0x7fe01814a700 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe018125f90 con 0x7fe01814a700 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 10 client.?.objecter ms_handle_connect 0x7fe0180b0770 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.057+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010871057 0 0) 0x7fe018125b40 con 0x7fe0180b0770 2024-03-29T11:22:56.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe018125cd0 con 0x7fe0180b0770 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1427985012 0 0) 0x7fe018125f90 con 0x7fe01814a700 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe018125b40 con 0x7fe01814a700 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe010001b30 con 0x7fe01814a700 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 10 monclient(hunting): dump: 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:56.044 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2834143853 0 0) 0x7fe004004eb0 con 0x7fe0180ac5a0 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe018125f90 con 0x7fe0180ac5a0 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe008001a10 con 0x7fe0180ac5a0 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.058+0000 7fdffffff640 10 monclient(hunting): dump: 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:56.045 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:56.046 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.046 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:56.046 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:56.046 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:56.046 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:56.046 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:56.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 721505246 0 0) 0x7fe018125cd0 con 0x7fe0180b0770 2024-03-29T11:22:56.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe004004eb0 con 0x7fe0180b0770 2024-03-29T11:22:56.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1110329382 0 0) 0x7fe018125f90 con 0x7fe0180ac5a0 2024-03-29T11:22:56.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 >> v1:172.21.15.67:6790/0 conn(0x7fe01814a700 legacy=0x7fe0181e5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 >> v1:172.21.15.82:6789/0 conn(0x7fe0180b0770 legacy=0x7fe018126bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fdffffff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:56.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fdffffff640 1 monclient: found mon.a 2024-03-29T11:22:56.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fdffffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:56.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fdffffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:56.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0181dc0d0 con 0x7fe0180ac5a0 2024-03-29T11:22:56.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fdffffff640 10 monclient: _finish_auth 0 2024-03-29T11:22:56.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fdffffff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:56.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fdffffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:56.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fe01df93640 5 monclient: authenticate success, global_id 4356 2024-03-29T11:22:56.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fe01df93640 10 monclient: _renew_subs 2024-03-29T11:22:56.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fe01df93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:56.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fe01df93640 1 -- 172.21.15.67:0/2345492090 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0181dc380 con 0x7fe0180ac5a0 2024-03-29T11:22:56.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fe01df93640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:56.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fe01df93640 10 monclient: _renew_subs 2024-03-29T11:22:56.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fe01df93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:56.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fe01df93640 1 -- 172.21.15.67:0/2345492090 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0181db870 con 0x7fe0180ac5a0 2024-03-29T11:22:56.047 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.059+0000 7fe01df93640 1 librados: init done 2024-03-29T11:22:56.049 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.061+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe008001b70 con 0x7fe0180ac5a0 2024-03-29T11:22:56.049 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.061+0000 7fdffffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:56.049 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.061+0000 7fe015ffb640 4 set_mon_vals no callback set 2024-03-29T11:22:56.049 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.061+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe008003a60 con 0x7fe0180ac5a0 2024-03-29T11:22:56.049 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.061+0000 7fdffffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.049 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.061+0000 7fdffffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:56.049 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.061+0000 7fdffffff640 10 monclient: dump: 2024-03-29T11:22:56.049 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:22:56.049 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.061+0000 7fe01df93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.061+0000 7fe01df93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.062+0000 7fe01df93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.062+0000 7fe01df93640 1 -- 172.21.15.67:0/2345492090 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff8005080 con 0x7fe0180ac5a0 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.064+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fe008003bc0 con 0x7fe0180ac5a0 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.065+0000 7fe016ffd640 10 monclient: get_auth_request con 0x7fe00406e170 auth_method 0 2024-03-29T11:22:56.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.066+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fe008084aa0 con 0x7fe0180ac5a0 2024-03-29T11:22:56.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.066+0000 7fdffffff640 10 client.4356.objecter ms_dispatch 0x7fe018000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:56.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.066+0000 7fdffffff640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:56.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.066+0000 7fdffffff640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:56.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.066+0000 7fdffffff640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-29T11:22:56.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.066+0000 7fdffffff640 10 client.4356.objecter ms_handle_connect 0x7fe00406e170 2024-03-29T11:22:56.052 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.075+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0080501b0 con 0x7fe0180ac5a0 2024-03-29T11:22:56.052 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.075+0000 7fdffffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:56.052 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.075+0000 7fdffffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:56.430 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.452+0000 7fe01df93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:56.430 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.452+0000 7fe01df93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:56.430 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.452+0000 7fe01df93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:22:56.430 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.452+0000 7fe01df93640 1 -- 172.21.15.67:0/2345492090 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdff8005330 con 0x7fe0180ac5a0 2024-03-29T11:22:56.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.454+0000 7fdffffff640 1 -- 172.21.15.67:0/2345492090 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fe0080522b0 con 0x7fe0180ac5a0 2024-03-29T11:22:56.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.454+0000 7fdffffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:56.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.454+0000 7fdffffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:56.432 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:22:56.432 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:56.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.457+0000 7fe01df93640 10 librados: watch_flush enter 2024-03-29T11:22:56.436 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.457+0000 7fe01df93640 10 librados: watch_flush exit 2024-03-29T11:22:56.436 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.457+0000 7fe01df93640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-29T11:22:56.436 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.457+0000 7fe01df93640 10 client.4356.objecter successfully canceled tick 2024-03-29T11:22:56.436 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.458+0000 7fe01df93640 1 -- 172.21.15.67:0/2345492090 >> v1:172.21.15.67:6816/404787197 conn(0x7fe00406e170 legacy=0x7fe004070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.436 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.458+0000 7fe01df93640 10 monclient: shutdown 2024-03-29T11:22:56.436 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.458+0000 7fe01df93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:56.436 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.458+0000 7fe01df93640 1 -- 172.21.15.67:0/2345492090 >> v1:172.21.15.67:6789/0 conn(0x7fe0180ac5a0 legacy=0x7fe0181264b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.436 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.458+0000 7fe01df93640 1 -- 172.21.15.67:0/2345492090 shutdown_connections 2024-03-29T11:22:56.436 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.458+0000 7fe01df93640 1 -- 172.21.15.67:0/2345492090 >> 172.21.15.67:0/2345492090 conn(0x7fe01806b660 msgr2=0x7fe0180abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:56.436 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.458+0000 7fe01df93640 1 -- 172.21.15.67:0/2345492090 shutdown_connections 2024-03-29T11:22:56.436 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.458+0000 7fe01df93640 1 -- 172.21.15.67:0/2345492090 wait complete. 2024-03-29T11:22:56.436 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:22:56.458+0000 7fe01df93640 1 librados: shutdown 2024-03-29T11:22:56.451 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:03.257 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:03.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.410+0000 7f5145e63640 10 monclient: get_monmap_and_config 2024-03-29T11:23:03.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.410+0000 7f5145e63640 10 monclient: build_initial_monmap 2024-03-29T11:23:03.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.410+0000 7f5145e63640 10 monclient: monmap: 2024-03-29T11:23:03.388 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:03.388 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:03.388 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:23:03.411164+0000 2024-03-29T11:23:03.388 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:23:03.411164+0000 2024-03-29T11:23:03.388 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:03.389 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:03.389 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:03.389 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:23:03.389 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:23:03.389 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:03.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.411+0000 7f5145e63640 1 Processor -- start 2024-03-29T11:23:03.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.411+0000 7f5145e63640 1 -- start start 2024-03-29T11:23:03.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.411+0000 7f5145e63640 10 monclient: init 2024-03-29T11:23:03.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.413+0000 7f5145e63640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:03.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.413+0000 7f5145e63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:03.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.413+0000 7f5145e63640 10 monclient(hunting): picked mon.noname-a con 0x7f51401457f0 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:03.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.413+0000 7f5145e63640 10 monclient(hunting): picked mon.noname-c con 0x7f514013f650 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:03.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.413+0000 7f5145e63640 10 monclient(hunting): picked mon.noname-b con 0x7f51400a9d70 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:03.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.413+0000 7f5145e63640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51400aaef0 con 0x7f51401457f0 2024-03-29T11:23:03.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.413+0000 7f5145e63640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5140145bd0 con 0x7f514013f650 2024-03-29T11:23:03.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.413+0000 7f5145e63640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5140145eb0 con 0x7f51400a9d70 2024-03-29T11:23:03.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.413+0000 7f5145e63640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:03.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.413+0000 7f5145e63640 10 monclient(hunting): authenticate will time out at 1043.709281s 2024-03-29T11:23:03.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.413+0000 7f513ffff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f51400a9d70 0x7f514014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:43788/0 (socket says 172.21.15.67:43788) 2024-03-29T11:23:03.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.413+0000 7f513ffff640 1 -- 172.21.15.67:0/1781009451 learned_addr learned my addr 172.21.15.67:0/1781009451 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:03.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.414+0000 7f513effd640 1 -- 172.21.15.67:0/1781009451 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141528111 0 0) 0x7f5140145eb0 con 0x7f51400a9d70 2024-03-29T11:23:03.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.414+0000 7f513effd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:03.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.414+0000 7f513effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:03.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.414+0000 7f513effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:03.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.414+0000 7f513effd640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:23:03.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.414+0000 7f513effd640 1 -- 172.21.15.67:0/1781009451 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f512c003520 con 0x7f51400a9d70 2024-03-29T11:23:03.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.414+0000 7f513effd640 1 -- 172.21.15.67:0/1781009451 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 746930948 0 0) 0x7f512c003520 con 0x7f51400a9d70 2024-03-29T11:23:03.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.414+0000 7f513effd640 1 -- 172.21.15.67:0/1781009451 >> v1:172.21.15.82:6789/0 conn(0x7f514013f650 legacy=0x7f51400b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:03.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.414+0000 7f513effd640 1 -- 172.21.15.67:0/1781009451 >> v1:172.21.15.67:6789/0 conn(0x7f51401457f0 legacy=0x7f514013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:03.396 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.416+0000 7f513effd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:03.396 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.416+0000 7f513effd640 1 monclient: found mon.noname-b 2024-03-29T11:23:03.396 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.416+0000 7f513effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:03.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.416+0000 7f513effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:23:03.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.416+0000 7f513effd640 1 -- 172.21.15.67:0/1781009451 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5140146340 con 0x7f51400a9d70 2024-03-29T11:23:03.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.416+0000 7f513effd640 10 monclient: _finish_auth 0 2024-03-29T11:23:03.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.416+0000 7f513effd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:03.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.416+0000 7f513effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:03.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.416+0000 7f513effd640 1 -- 172.21.15.67:0/1781009451 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5138001a10 con 0x7f51400a9d70 2024-03-29T11:23:03.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.416+0000 7f513effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:03.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.416+0000 7f513effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:03.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.416+0000 7f513effd640 10 monclient: dump: 2024-03-29T11:23:03.397 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:03.397 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:03.397 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:03.398 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:03.398 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:03.398 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:03.398 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:03.398 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:03.398 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:03.398 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:03.398 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.416+0000 7f5145e63640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:23:03.398 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.417+0000 7f5145e63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:03.398 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.417+0000 7f513effd640 1 -- 172.21.15.67:0/1781009451 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5138001ca0 con 0x7f51400a9d70 2024-03-29T11:23:03.398 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.417+0000 7f513effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:03.398 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.417+0000 7f5145e63640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:03.399 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.417+0000 7f5145e63640 4 set_mon_vals no callback set 2024-03-29T11:23:03.399 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.417+0000 7f513effd640 1 -- 172.21.15.67:0/1781009451 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5138002030 con 0x7f51400a9d70 2024-03-29T11:23:03.399 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.417+0000 7f513effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:03.399 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.417+0000 7f513effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:03.399 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.417+0000 7f513effd640 10 monclient: dump: 2024-03-29T11:23:03.399 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:03.399 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:03.399 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:03.399 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:03.399 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:03.399 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:03.400 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:03.400 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:03.400 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:03.400 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:03.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.418+0000 7f5145e63640 10 monclient: shutdown 2024-03-29T11:23:03.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.418+0000 7f5145e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:03.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.418+0000 7f5145e63640 1 -- 172.21.15.67:0/1781009451 >> v1:172.21.15.67:6790/0 conn(0x7f51400a9d70 legacy=0x7f514014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.418+0000 7f5145e63640 1 -- 172.21.15.67:0/1781009451 shutdown_connections 2024-03-29T11:23:03.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.418+0000 7f5145e63640 1 -- 172.21.15.67:0/1781009451 >> 172.21.15.67:0/1781009451 conn(0x7f5140066a00 msgr2=0x7f5140066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:03.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.419+0000 7f5145e63640 1 -- 172.21.15.67:0/1781009451 shutdown_connections 2024-03-29T11:23:03.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.419+0000 7f5145e63640 1 -- 172.21.15.67:0/1781009451 wait complete. 2024-03-29T11:23:03.406 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.420+0000 7f5145e63640 10 monclient: build_initial_monmap 2024-03-29T11:23:03.406 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.420+0000 7f5145e63640 10 monclient: monmap: 2024-03-29T11:23:03.406 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:03.406 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:03.406 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:23:03.406 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:23:03.406 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:03.407 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:03.407 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:03.407 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:23:03.407 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:23:03.407 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:03.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.420+0000 7f5145e63640 1 librados: starting msgr at 2024-03-29T11:23:03.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.420+0000 7f5145e63640 1 librados: starting objecter 2024-03-29T11:23:03.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.420+0000 7f5145e63640 1 Processor -- start 2024-03-29T11:23:03.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.421+0000 7f5145e63640 1 -- start start 2024-03-29T11:23:03.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.421+0000 7f5145e63640 1 librados: setting wanted keys 2024-03-29T11:23:03.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.421+0000 7f5145e63640 1 librados: calling monclient init 2024-03-29T11:23:03.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.421+0000 7f5145e63640 10 monclient: init 2024-03-29T11:23:03.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.422+0000 7f5145e63640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:03.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.422+0000 7f5145e63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:03.408 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.422+0000 7f5145e63640 10 monclient(hunting): picked mon.noname-c con 0x7f51400a9d70 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:03.408 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.422+0000 7f513f7fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f51400a9d70 0x7f51401cd390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:46646/0 (socket says 172.21.15.67:46646) 2024-03-29T11:23:03.408 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.422+0000 7f513f7fe640 1 -- 172.21.15.67:0/3040691797 learned_addr learned my addr 172.21.15.67:0/3040691797 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:03.408 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.422+0000 7f5145e63640 10 monclient(hunting): picked mon.noname-a con 0x7f514013f650 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:03.408 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.422+0000 7f513cff9640 10 client.?.objecter ms_handle_connect 0x7f51400a9d70 2024-03-29T11:23:03.408 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.422+0000 7f513cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:03.408 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.422+0000 7f5145e63640 10 monclient(hunting): picked mon.noname-b con 0x7f51401457f0 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:03.408 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.422+0000 7f5145e63640 1 -- 172.21.15.67:0/3040691797 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51401cb490 con 0x7f514013f650 2024-03-29T11:23:03.408 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.423+0000 7f5145e63640 1 -- 172.21.15.67:0/3040691797 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51401387a0 con 0x7f51401457f0 2024-03-29T11:23:03.408 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.423+0000 7f5145e63640 1 -- 172.21.15.67:0/3040691797 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51401cc9b0 con 0x7f51400a9d70 2024-03-29T11:23:03.408 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.423+0000 7f5145e63640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:03.408 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.423+0000 7f5145e63640 10 monclient(hunting): authenticate will time out at 1043.719186s 2024-03-29T11:23:03.409 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.423+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670374976 0 0) 0x7f51401cc9b0 con 0x7f51400a9d70 2024-03-29T11:23:03.409 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.423+0000 7f513cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:03.409 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.423+0000 7f513cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:03.409 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.423+0000 7f513cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:03.409 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.423+0000 7f513cff9640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:23:03.409 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.423+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5120004eb0 con 0x7f51400a9d70 2024-03-29T11:23:03.409 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.423+0000 7f513cff9640 10 client.?.objecter ms_handle_connect 0x7f51401457f0 2024-03-29T11:23:03.409 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.423+0000 7f513cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:03.409 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1325628340 0 0) 0x7f5120004eb0 con 0x7f51400a9d70 2024-03-29T11:23:03.409 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51401cc9b0 con 0x7f51400a9d70 2024-03-29T11:23:03.409 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990211515 0 0) 0x7f51401387a0 con 0x7f51401457f0 2024-03-29T11:23:03.409 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:03.409 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5120005590 con 0x7f51401457f0 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f51300017d0 con 0x7f51400a9d70 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 10 monclient(hunting): dump: 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:03.410 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:03.411 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:03.411 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:03.411 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:03.411 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 594733069 0 0) 0x7f51401cc9b0 con 0x7f51400a9d70 2024-03-29T11:23:03.411 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 >> v1:172.21.15.82:6789/0 conn(0x7f51401457f0 legacy=0x7f5140138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.411 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 >> v1:172.21.15.67:6789/0 conn(0x7f514013f650 legacy=0x7f5140137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.411 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:23:03.411 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 1 monclient: found mon.c 2024-03-29T11:23:03.411 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:03.411 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:03.411 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51401ccc90 con 0x7f51400a9d70 2024-03-29T11:23:03.411 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 10 monclient: _finish_auth 0 2024-03-29T11:23:03.411 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:23:03.411 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.424+0000 7f513cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.425+0000 7f5145e63640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.425+0000 7f5145e63640 10 monclient: _renew_subs 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.425+0000 7f5145e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.425+0000 7f5145e63640 1 -- 172.21.15.67:0/3040691797 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51401ccf70 con 0x7f51400a9d70 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.425+0000 7f5145e63640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.425+0000 7f5145e63640 10 monclient: _renew_subs 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.425+0000 7f5145e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.425+0000 7f5145e63640 1 -- 172.21.15.67:0/3040691797 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51401cbb00 con 0x7f51400a9d70 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.425+0000 7f5145e63640 1 librados: init done 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.425+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51300019f0 con 0x7f51400a9d70 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.425+0000 7f513cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.425+0000 7f513e7fc640 4 set_mon_vals no callback set 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.426+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5130004a80 con 0x7f51400a9d70 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.426+0000 7f513cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:03.412 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.426+0000 7f513cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:03.413 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.426+0000 7f513cff9640 10 monclient: dump: 2024-03-29T11:23:03.413 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:03.413 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:03.413 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:03.413 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:03.413 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:03.413 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:03.413 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:03.413 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:03.413 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:03.413 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:03.413 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.426+0000 7f5145e63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:03.413 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.426+0000 7f5145e63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:03.413 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.426+0000 7f5145e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:03.414 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.426+0000 7f5145e63640 1 -- 172.21.15.67:0/3040691797 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5124005080 con 0x7f51400a9d70 2024-03-29T11:23:03.414 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.427+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f513001aad0 con 0x7f51400a9d70 2024-03-29T11:23:03.414 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.428+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f5130084b10 con 0x7f51400a9d70 2024-03-29T11:23:03.414 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.428+0000 7f513cff9640 10 client.4370.objecter ms_dispatch 0x7f5140000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:03.414 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.428+0000 7f513cff9640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:03.414 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.428+0000 7f513cff9640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:03.414 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.428+0000 7f513cff9640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-29T11:23:03.414 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.428+0000 7f5136ffd640 10 monclient: get_auth_request con 0x7f512006df30 auth_method 0 2024-03-29T11:23:03.414 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.428+0000 7f513cff9640 10 client.4370.objecter ms_handle_connect 0x7f512006df30 2024-03-29T11:23:03.415 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.436+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5130050450 con 0x7f51400a9d70 2024-03-29T11:23:03.415 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.436+0000 7f513cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:03.415 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.436+0000 7f513cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:03.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.761+0000 7f5145e63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:03.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.761+0000 7f5145e63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:03.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.761+0000 7f5145e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:03.739 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.761+0000 7f5145e63640 1 -- 172.21.15.67:0/3040691797 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5124005330 con 0x7f51400a9d70 2024-03-29T11:23:03.740 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.762+0000 7f513cff9640 1 -- 172.21.15.67:0/3040691797 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f5130052380 con 0x7f51400a9d70 2024-03-29T11:23:03.740 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.762+0000 7f513cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:03.740 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.762+0000 7f513cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:03.740 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:23:03.740 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:03.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.765+0000 7f5145e63640 10 librados: watch_flush enter 2024-03-29T11:23:03.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.766+0000 7f5145e63640 10 librados: watch_flush exit 2024-03-29T11:23:03.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.766+0000 7f5145e63640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-29T11:23:03.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.766+0000 7f5145e63640 10 client.4370.objecter successfully canceled tick 2024-03-29T11:23:03.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.766+0000 7f5145e63640 1 -- 172.21.15.67:0/3040691797 >> v1:172.21.15.67:6816/404787197 conn(0x7f512006df30 legacy=0x7f51200703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.766+0000 7f5145e63640 10 monclient: shutdown 2024-03-29T11:23:03.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.766+0000 7f5145e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:03.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.766+0000 7f5145e63640 1 -- 172.21.15.67:0/3040691797 >> v1:172.21.15.67:6790/0 conn(0x7f51400a9d70 legacy=0x7f51401cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.766+0000 7f5145e63640 1 -- 172.21.15.67:0/3040691797 shutdown_connections 2024-03-29T11:23:03.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.766+0000 7f5145e63640 1 -- 172.21.15.67:0/3040691797 >> 172.21.15.67:0/3040691797 conn(0x7f5140066a00 msgr2=0x7f51400b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:03.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.766+0000 7f5145e63640 1 -- 172.21.15.67:0/3040691797 shutdown_connections 2024-03-29T11:23:03.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.766+0000 7f5145e63640 1 -- 172.21.15.67:0/3040691797 wait complete. 2024-03-29T11:23:03.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:03.766+0000 7f5145e63640 1 librados: shutdown 2024-03-29T11:23:03.759 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:10.567 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:10.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.716+0000 7fb8856b5640 10 monclient: get_monmap_and_config 2024-03-29T11:23:10.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.716+0000 7fb8856b5640 10 monclient: build_initial_monmap 2024-03-29T11:23:10.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.716+0000 7fb8856b5640 10 monclient: monmap: 2024-03-29T11:23:10.696 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:10.696 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:10.696 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:23:10.718045+0000 2024-03-29T11:23:10.696 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:23:10.718045+0000 2024-03-29T11:23:10.696 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:10.696 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:10.696 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:10.696 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:23:10.696 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:23:10.696 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:10.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.716+0000 7fb8856b5640 1 Processor -- start 2024-03-29T11:23:10.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.716+0000 7fb8856b5640 1 -- start start 2024-03-29T11:23:10.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.717+0000 7fb8856b5640 10 monclient: init 2024-03-29T11:23:10.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.718+0000 7fb8856b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:10.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.718+0000 7fb8856b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:10.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.718+0000 7fb8856b5640 10 monclient(hunting): picked mon.noname-c con 0x7fb8801457f0 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:10.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.718+0000 7fb8856b5640 10 monclient(hunting): picked mon.noname-b con 0x7fb88013f650 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:10.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.718+0000 7fb8856b5640 10 monclient(hunting): picked mon.noname-a con 0x7fb8800a9d70 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:10.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.718+0000 7fb8856b5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8800aaef0 con 0x7fb8800a9d70 2024-03-29T11:23:10.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.718+0000 7fb8856b5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb880145bd0 con 0x7fb8801457f0 2024-03-29T11:23:10.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.718+0000 7fb8856b5640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb880145eb0 con 0x7fb88013f650 2024-03-29T11:23:10.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.718+0000 7fb8856b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:10.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.718+0000 7fb8856b5640 10 monclient(hunting): authenticate will time out at 1051.015703s 2024-03-29T11:23:10.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.719+0000 7fb87e7fc640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fb88013f650 0x7fb8800b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:57164/0 (socket says 172.21.15.67:57164) 2024-03-29T11:23:10.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.719+0000 7fb87e7fc640 1 -- 172.21.15.67:0/90621603 learned_addr learned my addr 172.21.15.67:0/90621603 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:10.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.719+0000 7fb87dffb640 1 -- 172.21.15.67:0/90621603 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683255200 0 0) 0x7fb880145eb0 con 0x7fb88013f650 2024-03-29T11:23:10.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.719+0000 7fb87dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:10.700 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.719+0000 7fb87dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:10.700 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.719+0000 7fb87dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:10.700 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.719+0000 7fb87dffb640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:23:10.700 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.719+0000 7fb87dffb640 1 -- 172.21.15.67:0/90621603 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb868003520 con 0x7fb88013f650 2024-03-29T11:23:10.700 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.719+0000 7fb87dffb640 1 -- 172.21.15.67:0/90621603 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1095475822 0 0) 0x7fb868003520 con 0x7fb88013f650 2024-03-29T11:23:10.700 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.719+0000 7fb87dffb640 1 -- 172.21.15.67:0/90621603 >> v1:172.21.15.82:6789/0 conn(0x7fb8801457f0 legacy=0x7fb88013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:10.700 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.719+0000 7fb87dffb640 1 -- 172.21.15.67:0/90621603 >> v1:172.21.15.67:6789/0 conn(0x7fb8800a9d70 legacy=0x7fb88014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:10.702 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb87dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:10.702 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb87dffb640 1 monclient: found mon.noname-b 2024-03-29T11:23:10.702 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb87dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:10.702 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb87dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:23:10.702 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb87dffb640 1 -- 172.21.15.67:0/90621603 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb880146340 con 0x7fb88013f650 2024-03-29T11:23:10.702 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb87dffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:10.703 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb87dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:10.703 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb87dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:10.703 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb8856b5640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:23:10.703 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb8856b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:10.703 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb87dffb640 1 -- 172.21.15.67:0/90621603 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb874001690 con 0x7fb88013f650 2024-03-29T11:23:10.703 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb87dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:10.703 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb87dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:10.703 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb87dffb640 10 monclient: dump: 2024-03-29T11:23:10.703 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:10.703 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:10.703 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:10.704 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:10.704 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:10.704 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:10.704 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:10.704 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:10.704 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:10.704 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:10.704 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb8856b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:10.704 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb87dffb640 1 -- 172.21.15.67:0/90621603 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8740019a0 con 0x7fb88013f650 2024-03-29T11:23:10.704 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb87dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:10.704 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb8856b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:10.704 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.721+0000 7fb8856b5640 4 set_mon_vals no callback set 2024-03-29T11:23:10.704 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.722+0000 7fb87dffb640 1 -- 172.21.15.67:0/90621603 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb874001d30 con 0x7fb88013f650 2024-03-29T11:23:10.705 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.722+0000 7fb87dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:10.705 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.722+0000 7fb87dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:10.705 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.722+0000 7fb87dffb640 10 monclient: dump: 2024-03-29T11:23:10.705 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:10.705 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:10.705 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:10.705 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:10.705 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:10.705 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:10.705 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:10.705 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:10.705 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:10.705 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:10.705 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.722+0000 7fb8856b5640 10 monclient: shutdown 2024-03-29T11:23:10.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.722+0000 7fb8856b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:10.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.722+0000 7fb8856b5640 1 -- 172.21.15.67:0/90621603 >> v1:172.21.15.67:6790/0 conn(0x7fb88013f650 legacy=0x7fb8800b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.722+0000 7fb8856b5640 1 -- 172.21.15.67:0/90621603 shutdown_connections 2024-03-29T11:23:10.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.722+0000 7fb8856b5640 1 -- 172.21.15.67:0/90621603 >> 172.21.15.67:0/90621603 conn(0x7fb880066a00 msgr2=0x7fb880066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:10.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.722+0000 7fb8856b5640 1 -- 172.21.15.67:0/90621603 shutdown_connections 2024-03-29T11:23:10.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.722+0000 7fb8856b5640 1 -- 172.21.15.67:0/90621603 wait complete. 2024-03-29T11:23:10.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.723+0000 7fb8856b5640 10 monclient: build_initial_monmap 2024-03-29T11:23:10.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.723+0000 7fb8856b5640 10 monclient: monmap: 2024-03-29T11:23:10.706 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:10.706 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:10.706 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:23:10.707 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:23:10.707 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:10.707 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:10.707 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:10.707 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:23:10.707 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:23:10.707 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:10.713 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.724+0000 7fb8856b5640 1 librados: starting msgr at 2024-03-29T11:23:10.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.724+0000 7fb8856b5640 1 librados: starting objecter 2024-03-29T11:23:10.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.724+0000 7fb8856b5640 1 Processor -- start 2024-03-29T11:23:10.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.724+0000 7fb8856b5640 1 -- start start 2024-03-29T11:23:10.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.724+0000 7fb8856b5640 1 librados: setting wanted keys 2024-03-29T11:23:10.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.724+0000 7fb8856b5640 1 librados: calling monclient init 2024-03-29T11:23:10.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.724+0000 7fb8856b5640 10 monclient: init 2024-03-29T11:23:10.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.725+0000 7fb8856b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:10.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.725+0000 7fb8856b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:10.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.725+0000 7fb8856b5640 10 monclient(hunting): picked mon.noname-c con 0x7fb8800a9d70 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:10.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.725+0000 7fb8856b5640 10 monclient(hunting): picked mon.noname-b con 0x7fb88013f650 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:10.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.725+0000 7fb8856b5640 10 monclient(hunting): picked mon.noname-a con 0x7fb8801457f0 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:10.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.725+0000 7fb8856b5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb880141ed0 con 0x7fb8801457f0 2024-03-29T11:23:10.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.725+0000 7fb8856b5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8801401f0 con 0x7fb88013f650 2024-03-29T11:23:10.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.725+0000 7fb8856b5640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8801d49e0 con 0x7fb8800a9d70 2024-03-29T11:23:10.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.725+0000 7fb8856b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:10.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.725+0000 7fb8856b5640 10 monclient(hunting): authenticate will time out at 1051.022693s 2024-03-29T11:23:10.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.725+0000 7fb87e7fc640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fb88013f650 0x7fb8801d42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:45510/0 (socket says 172.21.15.67:45510) 2024-03-29T11:23:10.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.725+0000 7fb87e7fc640 1 -- 172.21.15.67:0/4196550010 learned_addr learned my addr 172.21.15.67:0/4196550010 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:10.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.726+0000 7fb8677fe640 10 client.?.objecter ms_handle_connect 0x7fb88013f650 2024-03-29T11:23:10.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.726+0000 7fb8677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:10.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.726+0000 7fb8677fe640 10 client.?.objecter ms_handle_connect 0x7fb8800a9d70 2024-03-29T11:23:10.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.726+0000 7fb8677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:10.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.726+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311166695 0 0) 0x7fb8801d49e0 con 0x7fb8800a9d70 2024-03-29T11:23:10.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.726+0000 7fb8677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:10.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.726+0000 7fb8677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:10.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.726+0000 7fb8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:10.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.726+0000 7fb8677fe640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:23:10.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.726+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb86c004eb0 con 0x7fb8800a9d70 2024-03-29T11:23:10.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463260368 0 0) 0x7fb8801401f0 con 0x7fb88013f650 2024-03-29T11:23:10.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:10.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:10.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:10.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:23:10.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8801d49e0 con 0x7fb88013f650 2024-03-29T11:23:10.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1484394388 0 0) 0x7fb86c004eb0 con 0x7fb8800a9d70 2024-03-29T11:23:10.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8801401f0 con 0x7fb8800a9d70 2024-03-29T11:23:10.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb8700017d0 con 0x7fb8800a9d70 2024-03-29T11:23:10.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:10.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:10.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 10 monclient(hunting): dump: 2024-03-29T11:23:10.716 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2404706628 0 0) 0x7fb8801d49e0 con 0x7fb88013f650 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb86c004eb0 con 0x7fb88013f650 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb874001650 con 0x7fb88013f650 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:10.717 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:10.718 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 10 monclient(hunting): dump: 2024-03-29T11:23:10.718 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:10.718 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:10.718 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:10.718 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:10.718 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:10.718 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:10.718 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:10.718 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:10.718 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:10.718 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:10.718 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4204182183 0 0) 0x7fb8801401f0 con 0x7fb8800a9d70 2024-03-29T11:23:10.718 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 >> v1:172.21.15.82:6789/0 conn(0x7fb88013f650 legacy=0x7fb8801d42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.718 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.727+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 >> v1:172.21.15.67:6789/0 conn(0x7fb8801457f0 legacy=0x7fb8801422f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:10.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.729+0000 7fb8677fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:10.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.729+0000 7fb8677fe640 1 monclient: found mon.c 2024-03-29T11:23:10.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.729+0000 7fb8677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:10.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.729+0000 7fb8677fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:10.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.729+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8801d4b70 con 0x7fb8800a9d70 2024-03-29T11:23:10.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.729+0000 7fb8677fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:10.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.729+0000 7fb8677fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:10.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.729+0000 7fb8677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:10.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.729+0000 7fb8856b5640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:23:10.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.729+0000 7fb8856b5640 10 monclient: _renew_subs 2024-03-29T11:23:10.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.729+0000 7fb8856b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:10.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.729+0000 7fb8856b5640 1 -- 172.21.15.67:0/4196550010 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88013fda0 con 0x7fb8800a9d70 2024-03-29T11:23:10.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.730+0000 7fb8856b5640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:10.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.730+0000 7fb8856b5640 10 monclient: _renew_subs 2024-03-29T11:23:10.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.730+0000 7fb8856b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:10.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.730+0000 7fb8856b5640 1 -- 172.21.15.67:0/4196550010 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb880142be0 con 0x7fb8800a9d70 2024-03-29T11:23:10.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.730+0000 7fb8856b5640 1 librados: init done 2024-03-29T11:23:10.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.730+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb870001d60 con 0x7fb8800a9d70 2024-03-29T11:23:10.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.730+0000 7fb8677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:10.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.730+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb870003790 con 0x7fb8800a9d70 2024-03-29T11:23:10.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.730+0000 7fb8677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:10.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.730+0000 7fb8677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:10.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.730+0000 7fb8677fe640 10 monclient: dump: 2024-03-29T11:23:10.720 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:10.720 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:10.720 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:10.720 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.730+0000 7fb87d7fa640 4 set_mon_vals no callback set 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.732+0000 7fb8856b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.732+0000 7fb8856b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.732+0000 7fb8856b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.732+0000 7fb8856b5640 1 -- 172.21.15.67:0/4196550010 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb860005080 con 0x7fb8800a9d70 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.732+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fb870019760 con 0x7fb8800a9d70 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.732+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fb8700837e0 con 0x7fb8800a9d70 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.732+0000 7fb8677fe640 10 client.4376.objecter ms_dispatch 0x7fb880000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.732+0000 7fb8677fe640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:10.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.732+0000 7fb8677fe640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:10.722 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.732+0000 7fb8677fe640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-29T11:23:10.722 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.733+0000 7fb87e7fc640 10 monclient: get_auth_request con 0x7fb86c06df20 auth_method 0 2024-03-29T11:23:10.722 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.733+0000 7fb8677fe640 10 client.4376.objecter ms_handle_connect 0x7fb86c06df20 2024-03-29T11:23:10.722 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.740+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb87004eef0 con 0x7fb8800a9d70 2024-03-29T11:23:10.722 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.740+0000 7fb8677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:10.723 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:10.740+0000 7fb8677fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:11.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.061+0000 7fb8856b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:11.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.062+0000 7fb8856b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:11.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.062+0000 7fb8856b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:11.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.062+0000 7fb8856b5640 1 -- 172.21.15.67:0/4196550010 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb860005330 con 0x7fb8800a9d70 2024-03-29T11:23:11.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.062+0000 7fb8677fe640 1 -- 172.21.15.67:0/4196550010 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fb870050ff0 con 0x7fb8800a9d70 2024-03-29T11:23:11.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.062+0000 7fb8677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:11.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.062+0000 7fb8677fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:11.042 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:23:11.042 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:11.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.066+0000 7fb8856b5640 10 librados: watch_flush enter 2024-03-29T11:23:11.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.066+0000 7fb8856b5640 10 librados: watch_flush exit 2024-03-29T11:23:11.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.066+0000 7fb8856b5640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-29T11:23:11.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.066+0000 7fb8856b5640 10 client.4376.objecter successfully canceled tick 2024-03-29T11:23:11.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.066+0000 7fb8856b5640 1 -- 172.21.15.67:0/4196550010 >> v1:172.21.15.67:6816/404787197 conn(0x7fb86c06df20 legacy=0x7fb86c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.066+0000 7fb8856b5640 10 monclient: shutdown 2024-03-29T11:23:11.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.066+0000 7fb8856b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:11.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.066+0000 7fb8856b5640 1 -- 172.21.15.67:0/4196550010 >> v1:172.21.15.67:6790/0 conn(0x7fb8800a9d70 legacy=0x7fb8801d6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.066+0000 7fb8856b5640 1 -- 172.21.15.67:0/4196550010 shutdown_connections 2024-03-29T11:23:11.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.066+0000 7fb8856b5640 1 -- 172.21.15.67:0/4196550010 >> 172.21.15.67:0/4196550010 conn(0x7fb880066a00 msgr2=0x7fb8800b48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:11.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.066+0000 7fb8856b5640 1 -- 172.21.15.67:0/4196550010 shutdown_connections 2024-03-29T11:23:11.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.066+0000 7fb8856b5640 1 -- 172.21.15.67:0/4196550010 wait complete. 2024-03-29T11:23:11.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:11.066+0000 7fb8856b5640 1 librados: shutdown 2024-03-29T11:23:11.059 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:17.865 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:18.011 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.033+0000 7f7ff7fff640 10 monclient: get_monmap_and_config 2024-03-29T11:23:18.012 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.033+0000 7f7ff7fff640 10 monclient: build_initial_monmap 2024-03-29T11:23:18.012 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.033+0000 7f7ff7fff640 10 monclient: monmap: 2024-03-29T11:23:18.012 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:18.012 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:18.012 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:23:18.034280+0000 2024-03-29T11:23:18.012 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:23:18.034280+0000 2024-03-29T11:23:18.012 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:18.012 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:18.012 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:18.012 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:23:18.012 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:23:18.012 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:18.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.034+0000 7f7ff7fff640 1 Processor -- start 2024-03-29T11:23:18.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.034+0000 7f7ff7fff640 1 -- start start 2024-03-29T11:23:18.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.034+0000 7f7ff7fff640 10 monclient: init 2024-03-29T11:23:18.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.035+0000 7f7ff7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:18.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.035+0000 7f7ff7fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:18.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.035+0000 7f7ff7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f7ff80b2800 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:18.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.035+0000 7f7ff7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff80a9c00 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:18.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.035+0000 7f7ff7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff80b38c0 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:18.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.035+0000 7f7ff7fff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8141eb0 con 0x7f7ff80b2800 2024-03-29T11:23:18.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.035+0000 7f7ff7fff640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80b22d0 con 0x7f7ff80a9c00 2024-03-29T11:23:18.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.035+0000 7f7ff7fff640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8142d30 con 0x7f7ff80b38c0 2024-03-29T11:23:18.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.035+0000 7f7ff7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:18.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.035+0000 7f7ff7fff640 10 monclient(hunting): authenticate will time out at 1058.331274s 2024-03-29T11:23:18.015 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.036+0000 7f7ff77fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f7ff80b38c0 0x7f7ff814d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:60076/0 (socket says 172.21.15.67:60076) 2024-03-29T11:23:18.015 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.036+0000 7f7ff77fe640 1 -- 172.21.15.67:0/1249798018 learned_addr learned my addr 172.21.15.67:0/1249798018 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:18.019 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 1 -- 172.21.15.67:0/1249798018 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206741360 0 0) 0x7f7ff8142d30 con 0x7f7ff80b38c0 2024-03-29T11:23:18.019 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:18.019 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:18.019 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:18.019 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:23:18.019 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 1 -- 172.21.15.67:0/1249798018 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe0003520 con 0x7f7ff80b38c0 2024-03-29T11:23:18.019 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 1 -- 172.21.15.67:0/1249798018 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111057521 0 0) 0x7f7ff8141eb0 con 0x7f7ff80b2800 2024-03-29T11:23:18.019 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:18.019 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 1 -- 172.21.15.67:0/1249798018 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff8142d30 con 0x7f7ff80b2800 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 1 -- 172.21.15.67:0/1249798018 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4206139088 0 0) 0x7f7fe0003520 con 0x7f7ff80b38c0 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 1 -- 172.21.15.67:0/1249798018 >> v1:172.21.15.82:6789/0 conn(0x7f7ff80a9c00 legacy=0x7f7ff813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 1 -- 172.21.15.67:0/1249798018 >> v1:172.21.15.67:6789/0 conn(0x7f7ff80b2800 legacy=0x7f7ff8142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 1 -- 172.21.15.67:0/1249798018 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff80aaf90 con 0x7f7ff80b38c0 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:18.020 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff7fff640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:23:18.021 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:18.021 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 1 -- 172.21.15.67:0/1249798018 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7ff0001a10 con 0x7f7ff80b38c0 2024-03-29T11:23:18.021 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:18.021 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:18.021 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 10 monclient: dump: 2024-03-29T11:23:18.021 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:18.021 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:18.021 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:18.021 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:18.021 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:18.021 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:18.021 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:18.021 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:18.021 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.039+0000 7f7ff5ffb640 1 -- 172.21.15.67:0/1249798018 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ff0001c30 con 0x7f7ff80b38c0 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.040+0000 7f7ff5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.040+0000 7f7ff7fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.040+0000 7f7ff7fff640 4 set_mon_vals no callback set 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.040+0000 7f7ff5ffb640 1 -- 172.21.15.67:0/1249798018 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7ff0001f40 con 0x7f7ff80b38c0 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.040+0000 7f7ff5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.040+0000 7f7ff5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.040+0000 7f7ff5ffb640 10 monclient: dump: 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:18.022 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:18.023 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:18.023 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:18.023 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:18.023 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:18.023 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:18.023 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.040+0000 7f7ff7fff640 10 monclient: shutdown 2024-03-29T11:23:18.023 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.040+0000 7f7ff7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:18.023 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.040+0000 7f7ff7fff640 1 -- 172.21.15.67:0/1249798018 >> v1:172.21.15.67:6790/0 conn(0x7f7ff80b38c0 legacy=0x7f7ff814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.023 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.041+0000 7f7ff7fff640 1 -- 172.21.15.67:0/1249798018 shutdown_connections 2024-03-29T11:23:18.023 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.041+0000 7f7ff7fff640 1 -- 172.21.15.67:0/1249798018 >> 172.21.15.67:0/1249798018 conn(0x7f7ff8066a00 msgr2=0x7f7ff8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:18.023 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.041+0000 7f7ff7fff640 1 -- 172.21.15.67:0/1249798018 shutdown_connections 2024-03-29T11:23:18.025 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.041+0000 7f7ff7fff640 1 -- 172.21.15.67:0/1249798018 wait complete. 2024-03-29T11:23:18.025 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.042+0000 7f7ff7fff640 10 monclient: build_initial_monmap 2024-03-29T11:23:18.025 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.042+0000 7f7ff7fff640 10 monclient: monmap: 2024-03-29T11:23:18.025 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:18.026 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:18.026 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:23:18.026 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:23:18.026 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:18.026 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:18.026 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:18.026 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:23:18.026 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:23:18.026 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:18.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.042+0000 7f7ff7fff640 1 librados: starting msgr at 2024-03-29T11:23:18.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.042+0000 7f7ff7fff640 1 librados: starting objecter 2024-03-29T11:23:18.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.042+0000 7f7ff7fff640 1 Processor -- start 2024-03-29T11:23:18.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.043+0000 7f7ff7fff640 1 -- start start 2024-03-29T11:23:18.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.043+0000 7f7ff7fff640 1 librados: setting wanted keys 2024-03-29T11:23:18.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.043+0000 7f7ff7fff640 1 librados: calling monclient init 2024-03-29T11:23:18.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.043+0000 7f7ff7fff640 10 monclient: init 2024-03-29T11:23:18.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.043+0000 7f7ff7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:18.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.043+0000 7f7ff7fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:18.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.043+0000 7f7ff7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff80a9c00 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:18.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.043+0000 7f7ff7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff80b2800 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:18.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.043+0000 7f7ff7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f7ff80b38c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:18.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.043+0000 7f7ff7fff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff813ec60 con 0x7f7ff80b38c0 2024-03-29T11:23:18.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.043+0000 7f7ff7fff640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8071da0 con 0x7f7ff80b2800 2024-03-29T11:23:18.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.043+0000 7f7ff7fff640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80ab5c0 con 0x7f7ff80a9c00 2024-03-29T11:23:18.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.043+0000 7f7ff7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:18.031 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.043+0000 7f7ff7fff640 10 monclient(hunting): authenticate will time out at 1058.339863s 2024-03-29T11:23:18.031 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.045+0000 7f7ff6ffd640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f7ff80a9c00 0x7f7ff806fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:38996/0 (socket says 172.21.15.67:38996) 2024-03-29T11:23:18.031 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.045+0000 7f7ff6ffd640 1 -- 172.21.15.67:0/3298474775 learned_addr learned my addr 172.21.15.67:0/3298474775 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:18.031 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.048+0000 7f7fdf7fe640 10 client.?.objecter ms_handle_connect 0x7f7ff80a9c00 2024-03-29T11:23:18.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.048+0000 7f7fdf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:18.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.048+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863855053 0 0) 0x7f7ff80ab5c0 con 0x7f7ff80a9c00 2024-03-29T11:23:18.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.048+0000 7f7fdf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:18.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.049+0000 7f7fdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:18.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.049+0000 7f7fdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:18.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.049+0000 7f7fdf7fe640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:23:18.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.049+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe4004eb0 con 0x7f7ff80a9c00 2024-03-29T11:23:18.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.049+0000 7f7fdf7fe640 10 client.?.objecter ms_handle_connect 0x7f7ff80b2800 2024-03-29T11:23:18.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.049+0000 7f7fdf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:18.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.050+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 660867444 0 0) 0x7f7fe4004eb0 con 0x7f7ff80a9c00 2024-03-29T11:23:18.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.050+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ff80ab5c0 con 0x7f7ff80a9c00 2024-03-29T11:23:18.033 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.050+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7fec001a90 con 0x7f7ff80a9c00 2024-03-29T11:23:18.033 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.050+0000 7f7fdf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:18.033 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.050+0000 7f7fdf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:18.033 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.050+0000 7f7fdf7fe640 10 monclient(hunting): dump: 2024-03-29T11:23:18.033 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:18.033 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:18.033 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:18.033 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:18.033 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:18.033 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:18.033 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:18.033 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:18.033 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:18.033 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:18.034 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.050+0000 7f7fdf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:18.034 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.050+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231105754 0 0) 0x7f7ff8071da0 con 0x7f7ff80b2800 2024-03-29T11:23:18.034 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.050+0000 7f7fdf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:18.034 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.050+0000 7f7fdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:18.034 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.050+0000 7f7fdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:18.034 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.050+0000 7f7fdf7fe640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:23:18.034 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.050+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe4004eb0 con 0x7f7ff80b2800 2024-03-29T11:23:18.034 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.051+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2458836360 0 0) 0x7f7ff80ab5c0 con 0x7f7ff80a9c00 2024-03-29T11:23:18.034 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.052+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 >> v1:172.21.15.82:6789/0 conn(0x7f7ff80b2800 legacy=0x7f7ff8070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.034 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.052+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 >> v1:172.21.15.67:6789/0 conn(0x7f7ff80b38c0 legacy=0x7f7ff81eec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.034 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.052+0000 7f7fdf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:18.034 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.052+0000 7f7fdf7fe640 1 monclient: found mon.c 2024-03-29T11:23:18.034 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.052+0000 7f7fdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:18.035 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.052+0000 7f7fdf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:18.035 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.052+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff8070c10 con 0x7f7ff80a9c00 2024-03-29T11:23:18.035 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.052+0000 7f7fdf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:18.035 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.052+0000 7f7fdf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:18.035 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.053+0000 7f7fdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:18.035 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.053+0000 7f7fdf7fe640 10 client.?.objecter ms_handle_connect 0x7f7ff80b38c0 2024-03-29T11:23:18.035 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.053+0000 7f7fdf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:18.035 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.053+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fec001bf0 con 0x7f7ff80a9c00 2024-03-29T11:23:18.035 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.053+0000 7f7fdf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:18.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.053+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7fec003b20 con 0x7f7ff80a9c00 2024-03-29T11:23:18.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.054+0000 7f7fdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:18.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.054+0000 7f7fdf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:18.037 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.054+0000 7f7fdf7fe640 10 monclient: dump: 2024-03-29T11:23:18.037 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:18.037 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:18.037 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:18.038 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:18.038 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:18.038 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:18.038 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:18.038 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:18.038 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:18.038 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:18.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.054+0000 7f7ff57fa640 4 set_mon_vals no callback set 2024-03-29T11:23:18.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.054+0000 7f7ff7fff640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:23:18.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.054+0000 7f7ff7fff640 10 monclient: _renew_subs 2024-03-29T11:23:18.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.054+0000 7f7ff7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:18.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.054+0000 7f7ff7fff640 1 -- 172.21.15.67:0/3298474775 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff813f120 con 0x7f7ff80a9c00 2024-03-29T11:23:18.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.055+0000 7f7ff7fff640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:18.039 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.055+0000 7f7ff7fff640 10 monclient: _renew_subs 2024-03-29T11:23:18.039 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.055+0000 7f7ff7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:18.039 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.055+0000 7f7ff7fff640 1 -- 172.21.15.67:0/3298474775 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff812e6d0 con 0x7f7ff80a9c00 2024-03-29T11:23:18.039 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.055+0000 7f7ff7fff640 1 librados: init done 2024-03-29T11:23:18.039 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.056+0000 7f7ff7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:18.039 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.056+0000 7f7ff7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:18.039 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.056+0000 7f7ff7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:18.039 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.057+0000 7f7ff7fff640 1 -- 172.21.15.67:0/3298474775 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff80b6ac0 con 0x7f7ff80a9c00 2024-03-29T11:23:18.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.059+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f7fec003c80 con 0x7f7ff80a9c00 2024-03-29T11:23:18.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.060+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f7fec084b10 con 0x7f7ff80a9c00 2024-03-29T11:23:18.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.060+0000 7f7fdf7fe640 10 client.4382.objecter ms_dispatch 0x7f7ff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:18.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.060+0000 7f7fdf7fe640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:18.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.060+0000 7f7fdf7fe640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:18.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.060+0000 7f7fdf7fe640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-29T11:23:18.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.060+0000 7f7ff67fc640 10 monclient: get_auth_request con 0x7f7fe406dd00 auth_method 0 2024-03-29T11:23:18.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.060+0000 7f7fdf7fe640 10 client.4382.objecter ms_handle_connect 0x7f7fe406dd00 2024-03-29T11:23:18.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.065+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fec050450 con 0x7f7ff80a9c00 2024-03-29T11:23:18.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.065+0000 7f7fdf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:18.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.065+0000 7f7fdf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.388+0000 7f7ff7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.388+0000 7f7ff7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.389+0000 7f7ff7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.389+0000 7f7ff7fff640 1 -- 172.21.15.67:0/3298474775 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ff813f2b0 con 0x7f7ff80a9c00 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.392+0000 7f7fdf7fe640 1 -- 172.21.15.67:0/3298474775 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f7fec052380 con 0x7f7ff80a9c00 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.392+0000 7f7fdf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.392+0000 7f7fdf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:18.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.396+0000 7f7ff7fff640 10 librados: watch_flush enter 2024-03-29T11:23:18.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.396+0000 7f7ff7fff640 10 librados: watch_flush exit 2024-03-29T11:23:18.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.396+0000 7f7ff7fff640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-29T11:23:18.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.396+0000 7f7ff7fff640 10 client.4382.objecter successfully canceled tick 2024-03-29T11:23:18.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.396+0000 7f7ff7fff640 1 -- 172.21.15.67:0/3298474775 >> v1:172.21.15.67:6816/404787197 conn(0x7f7fe406dd00 legacy=0x7f7fe40701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.396+0000 7f7ff7fff640 10 monclient: shutdown 2024-03-29T11:23:18.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.396+0000 7f7ff7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:18.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.396+0000 7f7ff7fff640 1 -- 172.21.15.67:0/3298474775 >> v1:172.21.15.67:6790/0 conn(0x7f7ff80a9c00 legacy=0x7f7ff806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.396+0000 7f7ff7fff640 1 -- 172.21.15.67:0/3298474775 shutdown_connections 2024-03-29T11:23:18.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.396+0000 7f7ff7fff640 1 -- 172.21.15.67:0/3298474775 >> 172.21.15.67:0/3298474775 conn(0x7f7ff8066a00 msgr2=0x7f7ff81414c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:18.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.397+0000 7f7ff7fff640 1 -- 172.21.15.67:0/3298474775 shutdown_connections 2024-03-29T11:23:18.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.397+0000 7f7ff7fff640 1 -- 172.21.15.67:0/3298474775 wait complete. 2024-03-29T11:23:18.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:18.397+0000 7f7ff7fff640 1 librados: shutdown 2024-03-29T11:23:18.392 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:25.199 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:25.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.370+0000 7f105c83b640 10 monclient: get_monmap_and_config 2024-03-29T11:23:25.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.370+0000 7f105c83b640 10 monclient: build_initial_monmap 2024-03-29T11:23:25.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.370+0000 7f105c83b640 10 monclient: monmap: 2024-03-29T11:23:25.350 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:25.350 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:25.350 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:23:25.371243+0000 2024-03-29T11:23:25.350 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:23:25.371243+0000 2024-03-29T11:23:25.350 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:25.350 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:25.350 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:25.350 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:23:25.350 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:23:25.351 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:25.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.371+0000 7f105c83b640 1 Processor -- start 2024-03-29T11:23:25.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.371+0000 7f105c83b640 1 -- start start 2024-03-29T11:23:25.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.371+0000 7f105c83b640 10 monclient: init 2024-03-29T11:23:25.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.372+0000 7f105c83b640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:25.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.372+0000 7f105c83b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:25.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.372+0000 7f105c83b640 10 monclient(hunting): picked mon.noname-c con 0x7f10580af390 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:25.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.372+0000 7f105c83b640 10 monclient(hunting): picked mon.noname-a con 0x7f10580b3990 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:25.353 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.372+0000 7f105c83b640 10 monclient(hunting): picked mon.noname-b con 0x7f10580b16d0 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:25.353 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.372+0000 7f105c83b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10580b47f0 con 0x7f10580b3990 2024-03-29T11:23:25.353 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.372+0000 7f105c83b640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10580b1b20 con 0x7f10580af390 2024-03-29T11:23:25.353 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.372+0000 7f105c83b640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10580b44a0 con 0x7f10580b16d0 2024-03-29T11:23:25.353 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.372+0000 7f105c83b640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:25.353 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.372+0000 7f105c83b640 10 monclient(hunting): authenticate will time out at 1065.668400s 2024-03-29T11:23:25.353 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.372+0000 7f1057fff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f10580b16d0 0x7f1058151d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:48314/0 (socket says 172.21.15.67:48314) 2024-03-29T11:23:25.353 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.372+0000 7f1057fff640 1 -- 172.21.15.67:0/2897869084 learned_addr learned my addr 172.21.15.67:0/2897869084 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:25.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.376+0000 7f10567fc640 1 -- 172.21.15.67:0/2897869084 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323115910 0 0) 0x7f10580b44a0 con 0x7f10580b16d0 2024-03-29T11:23:25.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.376+0000 7f10567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:25.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.376+0000 7f10567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:25.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.376+0000 7f10567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:25.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.376+0000 7f10567fc640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:23:25.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.376+0000 7f10567fc640 1 -- 172.21.15.67:0/2897869084 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1040003520 con 0x7f10580b16d0 2024-03-29T11:23:25.360 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 1 -- 172.21.15.67:0/2897869084 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266320462 0 0) 0x7f10580b1b20 con 0x7f10580af390 2024-03-29T11:23:25.360 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:25.360 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:25.360 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:25.360 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:23:25.361 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 1 -- 172.21.15.67:0/2897869084 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10580b44a0 con 0x7f10580af390 2024-03-29T11:23:25.361 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 1 -- 172.21.15.67:0/2897869084 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652279920 0 0) 0x7f10580b47f0 con 0x7f10580b3990 2024-03-29T11:23:25.361 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:25.361 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:25.361 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:25.361 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:23:25.361 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 1 -- 172.21.15.67:0/2897869084 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10580b1b20 con 0x7f10580b3990 2024-03-29T11:23:25.362 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 1 -- 172.21.15.67:0/2897869084 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 686017040 0 0) 0x7f1040003520 con 0x7f10580b16d0 2024-03-29T11:23:25.362 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 1 -- 172.21.15.67:0/2897869084 >> v1:172.21.15.82:6789/0 conn(0x7f10580af390 legacy=0x7f10581459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:25.362 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 1 -- 172.21.15.67:0/2897869084 >> v1:172.21.15.67:6789/0 conn(0x7f10580b3990 legacy=0x7f10580b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:25.362 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:25.362 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 1 monclient: found mon.noname-b 2024-03-29T11:23:25.362 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:25.362 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:23:25.362 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 1 -- 172.21.15.67:0/2897869084 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10580b4c80 con 0x7f10580b16d0 2024-03-29T11:23:25.363 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:25.363 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:25.363 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:25.363 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 1 -- 172.21.15.67:0/2897869084 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f1050001990 con 0x7f10580b16d0 2024-03-29T11:23:25.363 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:25.363 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:25.363 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f10567fc640 10 monclient: dump: 2024-03-29T11:23:25.363 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:25.364 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:25.364 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:25.364 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:25.364 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:25.364 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:25.364 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:25.364 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:25.364 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:25.364 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:25.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f105c83b640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:23:25.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.377+0000 7f105c83b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:25.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.378+0000 7f10567fc640 1 -- 172.21.15.67:0/2897869084 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1050001af0 con 0x7f10580b16d0 2024-03-29T11:23:25.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.378+0000 7f10567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:25.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.378+0000 7f10567fc640 1 -- 172.21.15.67:0/2897869084 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f1050001f40 con 0x7f10580b16d0 2024-03-29T11:23:25.365 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.378+0000 7f105c83b640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:25.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.378+0000 7f105c83b640 4 set_mon_vals no callback set 2024-03-29T11:23:25.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.378+0000 7f10567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:25.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.378+0000 7f10567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:25.366 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.378+0000 7f10567fc640 10 monclient: dump: 2024-03-29T11:23:25.366 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:25.366 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:25.366 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:25.367 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:25.367 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:25.367 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:25.367 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:25.367 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:25.367 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:25.367 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:25.367 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.379+0000 7f105c83b640 10 monclient: shutdown 2024-03-29T11:23:25.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.379+0000 7f105c83b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:25.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.379+0000 7f105c83b640 1 -- 172.21.15.67:0/2897869084 >> v1:172.21.15.67:6790/0 conn(0x7f10580b16d0 legacy=0x7f1058151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:25.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.379+0000 7f105c83b640 1 -- 172.21.15.67:0/2897869084 shutdown_connections 2024-03-29T11:23:25.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.379+0000 7f105c83b640 1 -- 172.21.15.67:0/2897869084 >> 172.21.15.67:0/2897869084 conn(0x7f105806d230 msgr2=0x7f1058066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:25.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.380+0000 7f105c83b640 1 -- 172.21.15.67:0/2897869084 shutdown_connections 2024-03-29T11:23:25.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.380+0000 7f105c83b640 1 -- 172.21.15.67:0/2897869084 wait complete. 2024-03-29T11:23:25.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.381+0000 7f105c83b640 10 monclient: build_initial_monmap 2024-03-29T11:23:25.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.381+0000 7f105c83b640 10 monclient: monmap: 2024-03-29T11:23:25.369 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:25.369 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:25.369 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:23:25.369 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:23:25.369 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:25.369 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:25.369 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:25.369 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:23:25.370 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:23:25.370 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:25.370 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.381+0000 7f105c83b640 1 librados: starting msgr at 2024-03-29T11:23:25.370 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.381+0000 7f105c83b640 1 librados: starting objecter 2024-03-29T11:23:25.370 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.382+0000 7f105c83b640 1 Processor -- start 2024-03-29T11:23:25.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.382+0000 7f105c83b640 1 -- start start 2024-03-29T11:23:25.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.382+0000 7f105c83b640 1 librados: setting wanted keys 2024-03-29T11:23:25.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.382+0000 7f105c83b640 1 librados: calling monclient init 2024-03-29T11:23:25.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.382+0000 7f105c83b640 10 monclient: init 2024-03-29T11:23:25.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.383+0000 7f105c83b640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:25.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.383+0000 7f105c83b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:25.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.383+0000 7f105c83b640 10 monclient(hunting): picked mon.noname-a con 0x7f10580af390 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:25.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.383+0000 7f105c83b640 10 monclient(hunting): picked mon.noname-c con 0x7f10580b16d0 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:25.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.383+0000 7f105c83b640 10 monclient(hunting): picked mon.noname-b con 0x7f10580b3990 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:25.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.383+0000 7f105c83b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105814a300 con 0x7f10580af390 2024-03-29T11:23:25.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.383+0000 7f105c83b640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105814a570 con 0x7f10580b3990 2024-03-29T11:23:25.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.383+0000 7f105c83b640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105806df30 con 0x7f10580b16d0 2024-03-29T11:23:25.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.383+0000 7f105c83b640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:25.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.383+0000 7f105c83b640 10 monclient(hunting): authenticate will time out at 1065.679152s 2024-03-29T11:23:25.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.384+0000 7f1056ffd640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f10580b16d0 0x7f10581410f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:51738/0 (socket says 172.21.15.67:51738) 2024-03-29T11:23:25.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.384+0000 7f1056ffd640 1 -- 172.21.15.67:0/823850249 learned_addr learned my addr 172.21.15.67:0/823850249 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:25.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 10 client.?.objecter ms_handle_connect 0x7f10580b16d0 2024-03-29T11:23:25.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:25.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689382813 0 0) 0x7f105806df30 con 0x7f10580b16d0 2024-03-29T11:23:25.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:25.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:25.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:25.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:23:25.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1044004eb0 con 0x7f10580b16d0 2024-03-29T11:23:25.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1616988705 0 0) 0x7f1044004eb0 con 0x7f10580b16d0 2024-03-29T11:23:25.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f105806df30 con 0x7f10580b16d0 2024-03-29T11:23:25.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f1048001970 con 0x7f10580b16d0 2024-03-29T11:23:25.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:25.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:25.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 10 monclient(hunting): dump: 2024-03-29T11:23:25.379 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:25.379 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:25.379 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:25.380 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:25.380 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:25.380 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:25.380 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:25.380 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:25.380 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:25.380 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:25.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.385+0000 7f103ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:25.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.386+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3396466855 0 0) 0x7f105806df30 con 0x7f10580b16d0 2024-03-29T11:23:25.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.386+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 >> v1:172.21.15.82:6789/0 conn(0x7f10580b3990 legacy=0x7f10581e6100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:25.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.387+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 >> v1:172.21.15.67:6789/0 conn(0x7f10580af390 legacy=0x7f10581409e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:25.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.389+0000 7f103ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:25.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.389+0000 7f103ffff640 1 monclient: found mon.c 2024-03-29T11:23:25.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.389+0000 7f103ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.389+0000 7f103ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.389+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1058148840 con 0x7f10580b16d0 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f103ffff640 10 monclient: _finish_auth 0 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f103ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f103ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f105c83b640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f105c83b640 10 monclient: _renew_subs 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f105c83b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f105c83b640 1 -- 172.21.15.67:0/823850249 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105814a850 con 0x7f10580b16d0 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f105c83b640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f105c83b640 10 monclient: _renew_subs 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f105c83b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f105c83b640 1 -- 172.21.15.67:0/823850249 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1058149270 con 0x7f10580b16d0 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f105c83b640 1 librados: init done 2024-03-29T11:23:25.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1048001c70 con 0x7f10580b16d0 2024-03-29T11:23:25.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f103ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:25.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.390+0000 7f1055ffb640 4 set_mon_vals no callback set 2024-03-29T11:23:25.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.391+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f1048003a60 con 0x7f10580b16d0 2024-03-29T11:23:25.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.391+0000 7f103ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:25.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.391+0000 7f103ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:25.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.391+0000 7f103ffff640 10 monclient: dump: 2024-03-29T11:23:25.382 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:25.382 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:25.382 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:25.382 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:25.382 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:25.382 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:25.383 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:25.383 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:25.383 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:25.383 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:25.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.391+0000 7f105c83b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:25.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.391+0000 7f105c83b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:25.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.391+0000 7f105c83b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:25.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.391+0000 7f105c83b640 1 -- 172.21.15.67:0/823850249 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1038005080 con 0x7f10580b16d0 2024-03-29T11:23:25.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.396+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f1048003bc0 con 0x7f10580b16d0 2024-03-29T11:23:25.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.397+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f1048084b00 con 0x7f10580b16d0 2024-03-29T11:23:25.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.398+0000 7f10577fe640 10 monclient: get_auth_request con 0x7f104406dbe0 auth_method 0 2024-03-29T11:23:25.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.398+0000 7f103ffff640 10 client.4388.objecter ms_dispatch 0x7f1058000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:25.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.398+0000 7f103ffff640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:25.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.398+0000 7f103ffff640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:25.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.398+0000 7f103ffff640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-29T11:23:25.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.398+0000 7f103ffff640 10 client.4388.objecter ms_handle_connect 0x7f104406dbe0 2024-03-29T11:23:25.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.404+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10480503e0 con 0x7f10580b16d0 2024-03-29T11:23:25.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.404+0000 7f103ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:25.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.404+0000 7f103ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.741+0000 7f105c83b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.741+0000 7f105c83b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.741+0000 7f105c83b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.741+0000 7f105c83b640 1 -- 172.21.15.67:0/823850249 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1038005330 con 0x7f10580b16d0 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.742+0000 7f103ffff640 1 -- 172.21.15.67:0/823850249 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f1048052310 con 0x7f10580b16d0 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.742+0000 7f103ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.742+0000 7f103ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:23:25.722 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.746+0000 7f105c83b640 10 librados: watch_flush enter 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.746+0000 7f105c83b640 10 librados: watch_flush exit 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.746+0000 7f105c83b640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.746+0000 7f105c83b640 10 client.4388.objecter successfully canceled tick 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.747+0000 7f105c83b640 1 -- 172.21.15.67:0/823850249 >> v1:172.21.15.67:6816/404787197 conn(0x7f104406dbe0 legacy=0x7f10440700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.747+0000 7f105c83b640 10 monclient: shutdown 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.747+0000 7f105c83b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.747+0000 7f105c83b640 1 -- 172.21.15.67:0/823850249 >> v1:172.21.15.67:6790/0 conn(0x7f10580b16d0 legacy=0x7f10581410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.747+0000 7f105c83b640 1 -- 172.21.15.67:0/823850249 shutdown_connections 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.747+0000 7f105c83b640 1 -- 172.21.15.67:0/823850249 >> 172.21.15.67:0/823850249 conn(0x7f105806d230 msgr2=0x7f10580b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.748+0000 7f105c83b640 1 -- 172.21.15.67:0/823850249 shutdown_connections 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.748+0000 7f105c83b640 1 -- 172.21.15.67:0/823850249 wait complete. 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:25.748+0000 7f105c83b640 1 librados: shutdown 2024-03-29T11:23:25.744 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:32.550 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:32.703 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.723+0000 7f5794257640 10 monclient: get_monmap_and_config 2024-03-29T11:23:32.703 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.723+0000 7f5794257640 10 monclient: build_initial_monmap 2024-03-29T11:23:32.703 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.723+0000 7f5794257640 10 monclient: monmap: 2024-03-29T11:23:32.703 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:32.703 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:32.703 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:23:32.724374+0000 2024-03-29T11:23:32.703 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:23:32.724374+0000 2024-03-29T11:23:32.703 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:32.703 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:32.704 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:32.704 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:23:32.704 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:23:32.704 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:32.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.725+0000 7f5794257640 1 Processor -- start 2024-03-29T11:23:32.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.726+0000 7f5794257640 1 -- start start 2024-03-29T11:23:32.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.726+0000 7f5794257640 10 monclient: init 2024-03-29T11:23:32.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.727+0000 7f5794257640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:32.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.727+0000 7f5794257640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:32.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.727+0000 7f5794257640 10 monclient(hunting): picked mon.noname-a con 0x7f57840389e0 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:32.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.727+0000 7f5794257640 10 monclient(hunting): picked mon.noname-c con 0x7f5784038dc0 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:32.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.727+0000 7f5794257640 10 monclient(hunting): picked mon.noname-b con 0x7f5784030660 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:32.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.727+0000 7f5794257640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57840c54c0 con 0x7f57840389e0 2024-03-29T11:23:32.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.727+0000 7f5794257640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5784036430 con 0x7f5784038dc0 2024-03-29T11:23:32.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.727+0000 7f5794257640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5784039840 con 0x7f5784030660 2024-03-29T11:23:32.707 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.727+0000 7f5794257640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:32.707 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.727+0000 7f5794257640 10 monclient(hunting): authenticate will time out at 1073.023100s 2024-03-29T11:23:32.710 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.731+0000 7f5791ff3640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f57840389e0 0x7f5784033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34473/0 (socket says 172.21.15.67:34473) 2024-03-29T11:23:32.710 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.731+0000 7f5791ff3640 1 -- 172.21.15.67:0/1750916983 learned_addr learned my addr 172.21.15.67:0/1750916983 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:32.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.731+0000 7f5790ff1640 1 -- 172.21.15.67:0/1750916983 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398450837 0 0) 0x7f57840c54c0 con 0x7f57840389e0 2024-03-29T11:23:32.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.731+0000 7f5790ff1640 10 monclient(hunting): _negotiate 2024-03-29T11:23:32.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.731+0000 7f5790ff1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:32.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.731+0000 7f5790ff1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:32.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.731+0000 7f5790ff1640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:23:32.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.731+0000 7f5790ff1640 1 -- 172.21.15.67:0/1750916983 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5778003520 con 0x7f57840389e0 2024-03-29T11:23:32.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 1 -- 172.21.15.67:0/1750916983 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3782977879 0 0) 0x7f5778003520 con 0x7f57840389e0 2024-03-29T11:23:32.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 1 -- 172.21.15.67:0/1750916983 >> v1:172.21.15.67:6790/0 conn(0x7f5784030660 legacy=0x7f57840d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 1 -- 172.21.15.67:0/1750916983 >> v1:172.21.15.82:6789/0 conn(0x7f5784038dc0 legacy=0x7f57840cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:32.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 10 monclient: _finish_hunting 0 2024-03-29T11:23:32.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 1 monclient: found mon.noname-a 2024-03-29T11:23:32.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:32.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:23:32.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 1 -- 172.21.15.67:0/1750916983 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5784035690 con 0x7f57840389e0 2024-03-29T11:23:32.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 10 monclient: _finish_auth 0 2024-03-29T11:23:32.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 10 monclient: _check_auth_tickets 2024-03-29T11:23:32.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:32.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 1 -- 172.21.15.67:0/1750916983 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f578c04f730 con 0x7f57840389e0 2024-03-29T11:23:32.716 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:32.717 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:32.717 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.735+0000 7f5790ff1640 10 monclient: dump: 2024-03-29T11:23:32.717 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:32.717 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:32.717 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:32.717 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:32.717 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:32.717 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:32.717 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:32.717 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:32.717 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:32.717 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:32.717 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.736+0000 7f5794257640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:23:32.718 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.736+0000 7f5794257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:32.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.738+0000 7f5790ff1640 1 -- 172.21.15.67:0/1750916983 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f578c075f10 con 0x7f57840389e0 2024-03-29T11:23:32.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.738+0000 7f5790ff1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:32.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.738+0000 7f5790ff1640 1 -- 172.21.15.67:0/1750916983 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f578c0758b0 con 0x7f57840389e0 2024-03-29T11:23:32.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.738+0000 7f5790ff1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:32.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.738+0000 7f5790ff1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:32.720 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.738+0000 7f5790ff1640 10 monclient: dump: 2024-03-29T11:23:32.720 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:32.720 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:32.720 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:32.720 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:32.720 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:32.720 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:32.721 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:32.721 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:32.721 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:32.721 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:32.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.738+0000 7f5794257640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:32.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.738+0000 7f5794257640 4 set_mon_vals no callback set 2024-03-29T11:23:32.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.739+0000 7f5794257640 10 monclient: shutdown 2024-03-29T11:23:32.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.739+0000 7f5794257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:32.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.739+0000 7f5794257640 1 -- 172.21.15.67:0/1750916983 >> v1:172.21.15.67:6789/0 conn(0x7f57840389e0 legacy=0x7f5784033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.739+0000 7f5794257640 1 -- 172.21.15.67:0/1750916983 shutdown_connections 2024-03-29T11:23:32.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.739+0000 7f5794257640 1 -- 172.21.15.67:0/1750916983 >> 172.21.15.67:0/1750916983 conn(0x7f5784005c00 msgr2=0x7f5784006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:32.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.740+0000 7f5794257640 1 -- 172.21.15.67:0/1750916983 shutdown_connections 2024-03-29T11:23:32.721 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.740+0000 7f5794257640 1 -- 172.21.15.67:0/1750916983 wait complete. 2024-03-29T11:23:32.722 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.740+0000 7f5794257640 10 monclient: build_initial_monmap 2024-03-29T11:23:32.722 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.740+0000 7f5794257640 10 monclient: monmap: 2024-03-29T11:23:32.722 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:32.722 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:32.722 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:23:32.722 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:23:32.722 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:32.722 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:32.722 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:32.722 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:23:32.722 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:23:32.722 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:32.723 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.741+0000 7f5794257640 1 librados: starting msgr at 2024-03-29T11:23:32.723 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.741+0000 7f5794257640 1 librados: starting objecter 2024-03-29T11:23:32.723 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.741+0000 7f5794257640 1 Processor -- start 2024-03-29T11:23:32.725 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.745+0000 7f5794257640 1 -- start start 2024-03-29T11:23:32.727 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.745+0000 7f5794257640 1 librados: setting wanted keys 2024-03-29T11:23:32.727 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.745+0000 7f5794257640 1 librados: calling monclient init 2024-03-29T11:23:32.727 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.745+0000 7f5794257640 10 monclient: init 2024-03-29T11:23:32.727 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.746+0000 7f5794257640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:32.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.746+0000 7f5794257640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:32.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.746+0000 7f5794257640 10 monclient(hunting): picked mon.noname-b con 0x7f5784030660 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:32.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.746+0000 7f5794257640 10 monclient(hunting): picked mon.noname-c con 0x7f57840389e0 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:32.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.746+0000 7f5794257640 10 monclient(hunting): picked mon.noname-a con 0x7f57840c84b0 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:32.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.746+0000 7f5794257640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57840c82d0 con 0x7f57840c84b0 2024-03-29T11:23:32.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.746+0000 7f5794257640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57840c2b50 con 0x7f5784030660 2024-03-29T11:23:32.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.746+0000 7f5794257640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57840c64b0 con 0x7f57840389e0 2024-03-29T11:23:32.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.746+0000 7f5794257640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:32.728 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.746+0000 7f5794257640 10 monclient(hunting): authenticate will time out at 1073.042357s 2024-03-29T11:23:32.730 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.751+0000 7f5791ff3640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f5784030660 0x7f57840c6f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:50449/0 (socket says 172.21.15.67:50449) 2024-03-29T11:23:32.730 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.751+0000 7f5791ff3640 1 -- 172.21.15.67:0/3741755514 learned_addr learned my addr 172.21.15.67:0/3741755514 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:32.743 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 10 client.?.objecter ms_handle_connect 0x7f5784030660 2024-03-29T11:23:32.743 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:32.743 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 10 client.?.objecter ms_handle_connect 0x7f57840389e0 2024-03-29T11:23:32.743 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:32.743 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 10 client.?.objecter ms_handle_connect 0x7f57840c84b0 2024-03-29T11:23:32.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:32.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25307294 0 0) 0x7f57840c2b50 con 0x7f5784030660 2024-03-29T11:23:32.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:32.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:32.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:32.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:23:32.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5770004eb0 con 0x7f5784030660 2024-03-29T11:23:32.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748458570 0 0) 0x7f57840c64b0 con 0x7f57840389e0 2024-03-29T11:23:32.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:32.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:32.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:32.744 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.755+0000 7f57827fc640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:23:32.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.756+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57840c2b50 con 0x7f57840389e0 2024-03-29T11:23:32.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.756+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501775511 0 0) 0x7f57840c82d0 con 0x7f57840c84b0 2024-03-29T11:23:32.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.756+0000 7f57827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:32.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.756+0000 7f57827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:32.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.756+0000 7f57827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:32.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.756+0000 7f57827fc640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:23:32.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.756+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57840c64b0 con 0x7f57840c84b0 2024-03-29T11:23:32.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.756+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2437451688 0 0) 0x7f5770004eb0 con 0x7f5784030660 2024-03-29T11:23:32.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.756+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57840c82d0 con 0x7f5784030660 2024-03-29T11:23:32.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.756+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2451491144 0 0) 0x7f57840c2b50 con 0x7f57840389e0 2024-03-29T11:23:32.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.756+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5770004eb0 con 0x7f57840389e0 2024-03-29T11:23:32.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f578c04f400 con 0x7f5784030660 2024-03-29T11:23:32.745 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:32.746 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:32.746 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 10 monclient(hunting): dump: 2024-03-29T11:23:32.746 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:32.746 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:32.746 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:32.746 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:32.746 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:32.746 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:32.746 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:32.746 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:32.746 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:32.746 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:32.746 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:32.746 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5788001680 con 0x7f57840389e0 2024-03-29T11:23:32.747 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:32.747 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:32.747 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 10 monclient(hunting): dump: 2024-03-29T11:23:32.747 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:32.747 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:32.747 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:32.747 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:32.747 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:32.747 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:32.747 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:32.747 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:32.747 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:32.748 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:32.748 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1489895882 0 0) 0x7f57840c82d0 con 0x7f5784030660 2024-03-29T11:23:32.748 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 >> v1:172.21.15.67:6790/0 conn(0x7f57840389e0 legacy=0x7f57840c1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.748 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 >> v1:172.21.15.67:6789/0 conn(0x7f57840c84b0 legacy=0x7f57840c2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.748 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:32.748 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 1 monclient: found mon.b 2024-03-29T11:23:32.748 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:32.748 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:23:32.748 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.757+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57840c6790 con 0x7f5784030660 2024-03-29T11:23:32.748 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f57827fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:32.748 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f57827fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:32.748 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f57827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:32.748 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f5794257640 5 monclient: authenticate success, global_id 4354 2024-03-29T11:23:32.748 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f5794257640 10 monclient: _renew_subs 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f5794257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f5794257640 1 -- 172.21.15.67:0/3741755514 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57840c7f60 con 0x7f5784030660 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f578c04f730 con 0x7f5784030660 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f57827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f578c067d90 con 0x7f5784030660 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f57827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f5794257640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f57827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f57827fc640 10 monclient: dump: 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:32.749 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:32.750 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:32.750 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:32.750 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:32.750 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f57837fe640 4 set_mon_vals no callback set 2024-03-29T11:23:32.750 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f5794257640 10 monclient: _renew_subs 2024-03-29T11:23:32.750 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f5794257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:23:32.750 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f5794257640 1 -- 172.21.15.67:0/3741755514 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57840ca790 con 0x7f5784030660 2024-03-29T11:23:32.750 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.758+0000 7f5794257640 1 librados: init done 2024-03-29T11:23:32.750 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.760+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f578c067ef0 con 0x7f5784030660 2024-03-29T11:23:32.750 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.761+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f578c0ff370 con 0x7f5784030660 2024-03-29T11:23:32.750 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.761+0000 7f57827fc640 10 client.4354.objecter ms_dispatch 0x7f578c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:32.750 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.761+0000 7f57827fc640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:32.750 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.761+0000 7f57827fc640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:32.750 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.761+0000 7f57827fc640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-29T11:23:32.751 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.761+0000 7f5794257640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:32.751 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.761+0000 7f5794257640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:32.751 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.761+0000 7f5794257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:23:32.751 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.761+0000 7f5794257640 1 -- 172.21.15.67:0/3741755514 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5784007b90 con 0x7f5784030660 2024-03-29T11:23:32.751 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.761+0000 7f57917f2640 10 monclient: get_auth_request con 0x7f577006e0e0 auth_method 0 2024-03-29T11:23:32.751 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.765+0000 7f57827fc640 10 client.4354.objecter ms_handle_connect 0x7f577006e0e0 2024-03-29T11:23:32.754 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.772+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f578c0cac50 con 0x7f5784030660 2024-03-29T11:23:32.754 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.772+0000 7f57827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:32.754 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:32.772+0000 7f57827fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:33.087 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:23:33.088 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:33.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.105+0000 7f5794257640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:33.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.105+0000 7f5794257640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:33.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.105+0000 7f5794257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:23:33.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.105+0000 7f5794257640 1 -- 172.21.15.67:0/3741755514 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f578403c090 con 0x7f5784030660 2024-03-29T11:23:33.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.108+0000 7f57827fc640 1 -- 172.21.15.67:0/3741755514 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f578c0ccb80 con 0x7f5784030660 2024-03-29T11:23:33.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.108+0000 7f57827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:33.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.108+0000 7f57827fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:33.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.112+0000 7f5794257640 10 librados: watch_flush enter 2024-03-29T11:23:33.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.112+0000 7f5794257640 10 librados: watch_flush exit 2024-03-29T11:23:33.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.112+0000 7f5794257640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-29T11:23:33.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.112+0000 7f5794257640 10 client.4354.objecter successfully canceled tick 2024-03-29T11:23:33.094 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.112+0000 7f5794257640 1 -- 172.21.15.67:0/3741755514 >> v1:172.21.15.67:6816/404787197 conn(0x7f577006e0e0 legacy=0x7f57700705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:33.094 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.112+0000 7f5794257640 10 monclient: shutdown 2024-03-29T11:23:33.094 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.112+0000 7f5794257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:33.094 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.112+0000 7f5794257640 1 -- 172.21.15.67:0/3741755514 >> v1:172.21.15.82:6789/0 conn(0x7f5784030660 legacy=0x7f57840c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:33.094 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.113+0000 7f5794257640 1 -- 172.21.15.67:0/3741755514 shutdown_connections 2024-03-29T11:23:33.094 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.113+0000 7f5794257640 1 -- 172.21.15.67:0/3741755514 >> 172.21.15.67:0/3741755514 conn(0x7f5784005c00 msgr2=0x7f57840c4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:33.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.113+0000 7f5794257640 1 -- 172.21.15.67:0/3741755514 shutdown_connections 2024-03-29T11:23:33.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.113+0000 7f5794257640 1 -- 172.21.15.67:0/3741755514 wait complete. 2024-03-29T11:23:33.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:33.113+0000 7f5794257640 1 librados: shutdown 2024-03-29T11:23:33.109 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:39.916 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:40.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.105+0000 7f8dd73c3640 10 monclient: get_monmap_and_config 2024-03-29T11:23:40.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.105+0000 7f8dd73c3640 10 monclient: build_initial_monmap 2024-03-29T11:23:40.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.105+0000 7f8dd73c3640 10 monclient: monmap: 2024-03-29T11:23:40.085 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:40.085 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:40.085 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:23:40.106634+0000 2024-03-29T11:23:40.086 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:23:40.106634+0000 2024-03-29T11:23:40.086 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:40.086 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:40.086 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:40.086 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:23:40.086 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:23:40.086 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:40.086 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.106+0000 7f8dd73c3640 1 Processor -- start 2024-03-29T11:23:40.086 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.106+0000 7f8dd73c3640 1 -- start start 2024-03-29T11:23:40.086 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.106+0000 7f8dd73c3640 10 monclient: init 2024-03-29T11:23:40.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.107+0000 7f8dd73c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:40.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.107+0000 7f8dd73c3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:40.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.107+0000 7f8dd73c3640 10 monclient(hunting): picked mon.noname-b con 0x7f8dd00b2800 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:40.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.107+0000 7f8dd73c3640 10 monclient(hunting): picked mon.noname-a con 0x7f8dd00a9c00 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:40.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.107+0000 7f8dd73c3640 10 monclient(hunting): picked mon.noname-c con 0x7f8dd00b38c0 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:40.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.107+0000 7f8dd73c3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd0141eb0 con 0x7f8dd00a9c00 2024-03-29T11:23:40.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.107+0000 7f8dd73c3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd00b22d0 con 0x7f8dd00b38c0 2024-03-29T11:23:40.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.107+0000 7f8dd73c3640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd0142d30 con 0x7f8dd00b2800 2024-03-29T11:23:40.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.107+0000 7f8dd73c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:40.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.107+0000 7f8dd73c3640 10 monclient(hunting): authenticate will time out at 1080.403755s 2024-03-29T11:23:40.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.111+0000 7f8dd63c1640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f8dd00b2800 0x7f8dd0142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:49698/0 (socket says 172.21.15.67:49698) 2024-03-29T11:23:40.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.111+0000 7f8dd63c1640 1 -- 172.21.15.67:0/2645053407 learned_addr learned my addr 172.21.15.67:0/2645053407 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:40.091 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.112+0000 7f8dd53bf640 1 -- 172.21.15.67:0/2645053407 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259728796 0 0) 0x7f8dd0142d30 con 0x7f8dd00b2800 2024-03-29T11:23:40.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.112+0000 7f8dd53bf640 10 monclient(hunting): _negotiate 2024-03-29T11:23:40.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.112+0000 7f8dd53bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:40.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.112+0000 7f8dd53bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:40.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.112+0000 7f8dd53bf640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:23:40.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.112+0000 7f8dd53bf640 1 -- 172.21.15.67:0/2645053407 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dbc003520 con 0x7f8dd00b2800 2024-03-29T11:23:40.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.113+0000 7f8dd53bf640 1 -- 172.21.15.67:0/2645053407 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4082330695 0 0) 0x7f8dbc003520 con 0x7f8dd00b2800 2024-03-29T11:23:40.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.113+0000 7f8dd53bf640 1 -- 172.21.15.67:0/2645053407 >> v1:172.21.15.82:6789/0 conn(0x7f8dd00b38c0 legacy=0x7f8dd014d7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:40.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.115+0000 7f8dd53bf640 1 -- 172.21.15.67:0/2645053407 >> v1:172.21.15.67:6789/0 conn(0x7f8dd00a9c00 legacy=0x7f8dd013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:40.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.115+0000 7f8dd53bf640 10 monclient: _finish_hunting 0 2024-03-29T11:23:40.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.115+0000 7f8dd53bf640 1 monclient: found mon.noname-b 2024-03-29T11:23:40.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.115+0000 7f8dd53bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:40.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.115+0000 7f8dd53bf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:23:40.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.115+0000 7f8dd53bf640 1 -- 172.21.15.67:0/2645053407 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd00aaf90 con 0x7f8dd00b2800 2024-03-29T11:23:40.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.115+0000 7f8dd53bf640 10 monclient: _finish_auth 0 2024-03-29T11:23:40.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.115+0000 7f8dd53bf640 10 monclient: _check_auth_tickets 2024-03-29T11:23:40.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.115+0000 7f8dd53bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:40.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.116+0000 7f8dd73c3640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:23:40.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.116+0000 7f8dd73c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:40.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.116+0000 7f8dd53bf640 1 -- 172.21.15.67:0/2645053407 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f8dc8001a10 con 0x7f8dd00b2800 2024-03-29T11:23:40.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.116+0000 7f8dd53bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:40.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.116+0000 7f8dd53bf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:40.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.116+0000 7f8dd53bf640 10 monclient: dump: 2024-03-29T11:23:40.096 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:40.096 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:40.096 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:40.096 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:40.096 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:40.096 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:40.096 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:40.097 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:40.097 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:40.097 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:40.097 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.116+0000 7f8dd73c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:40.097 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.116+0000 7f8dd53bf640 1 -- 172.21.15.67:0/2645053407 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dc8001c30 con 0x7f8dd00b2800 2024-03-29T11:23:40.098 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.116+0000 7f8dd53bf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:40.098 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.116+0000 7f8dd73c3640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:40.098 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.116+0000 7f8dd73c3640 4 set_mon_vals no callback set 2024-03-29T11:23:40.098 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.117+0000 7f8dd53bf640 1 -- 172.21.15.67:0/2645053407 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f8dc8001fc0 con 0x7f8dd00b2800 2024-03-29T11:23:40.098 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.117+0000 7f8dd53bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:40.098 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.117+0000 7f8dd53bf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:40.098 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.117+0000 7f8dd53bf640 10 monclient: dump: 2024-03-29T11:23:40.098 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:40.098 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:40.098 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:40.098 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:40.098 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:40.098 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:40.099 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:40.099 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:40.099 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:40.099 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:40.099 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.117+0000 7f8dd73c3640 10 monclient: shutdown 2024-03-29T11:23:40.099 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.117+0000 7f8dd73c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:40.099 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.117+0000 7f8dd73c3640 1 -- 172.21.15.67:0/2645053407 >> v1:172.21.15.67:6790/0 conn(0x7f8dd00b2800 legacy=0x7f8dd0142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.099 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.117+0000 7f8dd73c3640 1 -- 172.21.15.67:0/2645053407 shutdown_connections 2024-03-29T11:23:40.099 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.117+0000 7f8dd73c3640 1 -- 172.21.15.67:0/2645053407 >> 172.21.15.67:0/2645053407 conn(0x7f8dd0066a00 msgr2=0x7f8dd0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:40.100 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.120+0000 7f8dd73c3640 1 -- 172.21.15.67:0/2645053407 shutdown_connections 2024-03-29T11:23:40.102 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.121+0000 7f8dd73c3640 1 -- 172.21.15.67:0/2645053407 wait complete. 2024-03-29T11:23:40.102 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.122+0000 7f8dd73c3640 10 monclient: build_initial_monmap 2024-03-29T11:23:40.102 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.122+0000 7f8dd73c3640 10 monclient: monmap: 2024-03-29T11:23:40.102 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:40.102 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:40.102 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:23:40.102 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:23:40.102 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:40.102 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:40.102 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:40.102 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:23:40.102 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:23:40.102 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:40.103 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.122+0000 7f8dd73c3640 1 librados: starting msgr at 2024-03-29T11:23:40.103 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.122+0000 7f8dd73c3640 1 librados: starting objecter 2024-03-29T11:23:40.103 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.122+0000 7f8dd73c3640 1 Processor -- start 2024-03-29T11:23:40.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.124+0000 7f8dd73c3640 1 -- start start 2024-03-29T11:23:40.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.124+0000 7f8dd73c3640 1 librados: setting wanted keys 2024-03-29T11:23:40.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.124+0000 7f8dd73c3640 1 librados: calling monclient init 2024-03-29T11:23:40.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.124+0000 7f8dd73c3640 10 monclient: init 2024-03-29T11:23:40.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.125+0000 7f8dd73c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:40.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.125+0000 7f8dd73c3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:40.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.125+0000 7f8dd73c3640 10 monclient(hunting): picked mon.noname-c con 0x7f8dd00a9c00 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:40.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.125+0000 7f8dd73c3640 10 monclient(hunting): picked mon.noname-a con 0x7f8dd00b2800 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:40.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.125+0000 7f8dd73c3640 10 monclient(hunting): picked mon.noname-b con 0x7f8dd00b38c0 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:40.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.125+0000 7f8dd73c3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd013f300 con 0x7f8dd00b2800 2024-03-29T11:23:40.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.125+0000 7f8dd73c3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd013f020 con 0x7f8dd00b38c0 2024-03-29T11:23:40.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.125+0000 7f8dd73c3640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd013edf0 con 0x7f8dd00a9c00 2024-03-29T11:23:40.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.125+0000 7f8dd73c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:40.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.125+0000 7f8dd73c3640 10 monclient(hunting): authenticate will time out at 1080.421623s 2024-03-29T11:23:40.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.129+0000 7f8dd63c1640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f8dd00a9c00 0x7f8dd013e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:52592/0 (socket says 172.21.15.67:52592) 2024-03-29T11:23:40.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.129+0000 7f8dd63c1640 1 -- 172.21.15.67:0/3636687815 learned_addr learned my addr 172.21.15.67:0/3636687815 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:40.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 client.?.objecter ms_handle_connect 0x7f8dd00b2800 2024-03-29T11:23:40.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:40.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 client.?.objecter ms_handle_connect 0x7f8dd00a9c00 2024-03-29T11:23:40.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:40.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042922214 0 0) 0x7f8dd013edf0 con 0x7f8dd00a9c00 2024-03-29T11:23:40.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:40.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:40.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:40.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:23:40.113 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db4004eb0 con 0x7f8dd00a9c00 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1660776278 0 0) 0x7f8db4004eb0 con 0x7f8dd00a9c00 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8dd013edf0 con 0x7f8dd00a9c00 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f8dc80018d0 con 0x7f8dd00a9c00 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:40.114 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:40.115 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:40.115 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:40.115 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895959666 0 0) 0x7f8dd013f300 con 0x7f8dd00b2800 2024-03-29T11:23:40.115 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:40.115 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:40.115 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:40.115 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:23:40.115 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.131+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db4004eb0 con 0x7f8dd00b2800 2024-03-29T11:23:40.115 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3870042664 0 0) 0x7f8dd013edf0 con 0x7f8dd00a9c00 2024-03-29T11:23:40.115 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 >> v1:172.21.15.82:6789/0 conn(0x7f8dd00b38c0 legacy=0x7f8dd013c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.115 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 >> v1:172.21.15.67:6789/0 conn(0x7f8dd00b2800 legacy=0x7f8dd01d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.115 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:40.115 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 1 monclient: found mon.c 2024-03-29T11:23:40.115 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd006de90 con 0x7f8dd00a9c00 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dc8004f70 con 0x7f8dd00a9c00 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f8dc8005280 con 0x7f8dd00a9c00 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc6ffd640 10 monclient: dump: 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:40.116 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.132+0000 7f8dc7fff640 4 set_mon_vals no callback set 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.133+0000 7f8dd73c3640 5 monclient: authenticate success, global_id 4400 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.133+0000 7f8dd73c3640 10 monclient: _renew_subs 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.133+0000 7f8dd73c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.133+0000 7f8dd73c3640 1 -- 172.21.15.67:0/3636687815 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd006e0e0 con 0x7f8dd00a9c00 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.133+0000 7f8dd73c3640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.133+0000 7f8dd73c3640 10 monclient: _renew_subs 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.133+0000 7f8dd73c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.133+0000 7f8dd73c3640 1 -- 172.21.15.67:0/3636687815 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd01d7fe0 con 0x7f8dd00a9c00 2024-03-29T11:23:40.117 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.133+0000 7f8dd73c3640 1 librados: init done 2024-03-29T11:23:40.119 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.135+0000 7f8dd73c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:40.119 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.135+0000 7f8dd73c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:40.119 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.135+0000 7f8dd73c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:40.119 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.135+0000 7f8dd73c3640 1 -- 172.21.15.67:0/3636687815 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd00b15f0 con 0x7f8dd00a9c00 2024-03-29T11:23:40.119 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.137+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f8dc801b250 con 0x7f8dd00a9c00 2024-03-29T11:23:40.119 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.137+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f8dc8085030 con 0x7f8dd00a9c00 2024-03-29T11:23:40.119 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.138+0000 7f8dc6ffd640 10 client.4400.objecter ms_dispatch 0x7f8dd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:40.119 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.138+0000 7f8dc6ffd640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:40.119 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.138+0000 7f8dc6ffd640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:40.119 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.138+0000 7f8dc6ffd640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-29T11:23:40.120 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.140+0000 7f8dd5bc0640 10 monclient: get_auth_request con 0x7f8db406dd70 auth_method 0 2024-03-29T11:23:40.120 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.141+0000 7f8dc6ffd640 10 client.4400.objecter ms_handle_connect 0x7f8db406dd70 2024-03-29T11:23:40.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.145+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8dc8050910 con 0x7f8dd00a9c00 2024-03-29T11:23:40.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.145+0000 7f8dc6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:40.126 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.145+0000 7f8dc6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:40.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.521+0000 7f8dd73c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:40.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.521+0000 7f8dd73c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:40.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.521+0000 7f8dd73c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:40.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.521+0000 7f8dd73c3640 1 -- 172.21.15.67:0/3636687815 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8dd004f730 con 0x7f8dd00a9c00 2024-03-29T11:23:40.504 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.525+0000 7f8dc6ffd640 1 -- 172.21.15.67:0/3636687815 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f8dc8052840 con 0x7f8dd00a9c00 2024-03-29T11:23:40.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.525+0000 7f8dc6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:40.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.525+0000 7f8dc6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:40.505 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:23:40.505 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:40.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.530+0000 7f8dd73c3640 10 librados: watch_flush enter 2024-03-29T11:23:40.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.530+0000 7f8dd73c3640 10 librados: watch_flush exit 2024-03-29T11:23:40.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.530+0000 7f8dd73c3640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-29T11:23:40.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.530+0000 7f8dd73c3640 10 client.4400.objecter successfully canceled tick 2024-03-29T11:23:40.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.530+0000 7f8dd73c3640 1 -- 172.21.15.67:0/3636687815 >> v1:172.21.15.67:6816/404787197 conn(0x7f8db406dd70 legacy=0x7f8db4070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.530+0000 7f8dd73c3640 10 monclient: shutdown 2024-03-29T11:23:40.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.530+0000 7f8dd73c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:40.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.530+0000 7f8dd73c3640 1 -- 172.21.15.67:0/3636687815 >> v1:172.21.15.67:6790/0 conn(0x7f8dd00a9c00 legacy=0x7f8dd013e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.531+0000 7f8dd73c3640 1 -- 172.21.15.67:0/3636687815 shutdown_connections 2024-03-29T11:23:40.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.531+0000 7f8dd73c3640 1 -- 172.21.15.67:0/3636687815 >> 172.21.15.67:0/3636687815 conn(0x7f8dd0066a00 msgr2=0x7f8dd00b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:40.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.531+0000 7f8dd73c3640 1 -- 172.21.15.67:0/3636687815 shutdown_connections 2024-03-29T11:23:40.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.531+0000 7f8dd73c3640 1 -- 172.21.15.67:0/3636687815 wait complete. 2024-03-29T11:23:40.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:40.532+0000 7f8dd73c3640 1 librados: shutdown 2024-03-29T11:23:40.530 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:47.337 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:47.488 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.508+0000 7f202bc0d640 10 monclient: get_monmap_and_config 2024-03-29T11:23:47.488 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.508+0000 7f202bc0d640 10 monclient: build_initial_monmap 2024-03-29T11:23:47.488 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.508+0000 7f202bc0d640 10 monclient: monmap: 2024-03-29T11:23:47.488 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:47.488 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:47.488 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:23:47.509341+0000 2024-03-29T11:23:47.489 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:23:47.509341+0000 2024-03-29T11:23:47.489 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:47.489 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:47.489 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:47.489 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:23:47.489 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:23:47.489 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:47.489 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.509+0000 7f202bc0d640 1 Processor -- start 2024-03-29T11:23:47.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.509+0000 7f202bc0d640 1 -- start start 2024-03-29T11:23:47.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.509+0000 7f202bc0d640 10 monclient: init 2024-03-29T11:23:47.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.510+0000 7f202bc0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:47.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.510+0000 7f202bc0d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:47.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.510+0000 7f202bc0d640 10 monclient(hunting): picked mon.noname-b con 0x7f20240b0770 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:47.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.510+0000 7f202bc0d640 10 monclient(hunting): picked mon.noname-c con 0x7f202414a700 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:47.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.510+0000 7f202bc0d640 10 monclient(hunting): picked mon.noname-a con 0x7f20240ac5a0 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:47.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.510+0000 7f202bc0d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20240aba70 con 0x7f20240ac5a0 2024-03-29T11:23:47.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.510+0000 7f202bc0d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20240aecb0 con 0x7f202414a700 2024-03-29T11:23:47.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.510+0000 7f202bc0d640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20240af150 con 0x7f20240b0770 2024-03-29T11:23:47.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.510+0000 7f202bc0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:47.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.510+0000 7f202bc0d640 10 monclient(hunting): authenticate will time out at 1087.806657s 2024-03-29T11:23:47.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.510+0000 7f20299a9640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f20240b0770 0x7f2024149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:35082/0 (socket says 172.21.15.67:35082) 2024-03-29T11:23:47.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.510+0000 7f20299a9640 1 -- 172.21.15.67:0/3375453187 learned_addr learned my addr 172.21.15.67:0/3375453187 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:47.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.511+0000 7f20289a7640 1 -- 172.21.15.67:0/3375453187 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626840330 0 0) 0x7f20240aba70 con 0x7f20240ac5a0 2024-03-29T11:23:47.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.511+0000 7f20289a7640 10 monclient(hunting): _negotiate 2024-03-29T11:23:47.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.511+0000 7f20289a7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:47.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.511+0000 7f20289a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:47.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.511+0000 7f20289a7640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:23:47.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.511+0000 7f20289a7640 1 -- 172.21.15.67:0/3375453187 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2010003520 con 0x7f20240ac5a0 2024-03-29T11:23:47.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 1 -- 172.21.15.67:0/3375453187 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954560016 0 0) 0x7f20240af150 con 0x7f20240b0770 2024-03-29T11:23:47.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 10 monclient(hunting): _negotiate 2024-03-29T11:23:47.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:47.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:47.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:23:47.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 1 -- 172.21.15.67:0/3375453187 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20240aba70 con 0x7f20240b0770 2024-03-29T11:23:47.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 1 -- 172.21.15.67:0/3375453187 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 706738772 0 0) 0x7f2010003520 con 0x7f20240ac5a0 2024-03-29T11:23:47.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 1 -- 172.21.15.67:0/3375453187 >> v1:172.21.15.67:6790/0 conn(0x7f20240b0770 legacy=0x7f2024149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:47.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 1 -- 172.21.15.67:0/3375453187 >> v1:172.21.15.82:6789/0 conn(0x7f202414a700 legacy=0x7f20240abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:47.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 10 monclient: _finish_hunting 0 2024-03-29T11:23:47.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 1 monclient: found mon.noname-a 2024-03-29T11:23:47.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:47.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:23:47.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 1 -- 172.21.15.67:0/3375453187 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20240ac9d0 con 0x7f20240ac5a0 2024-03-29T11:23:47.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 10 monclient: _finish_auth 0 2024-03-29T11:23:47.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 10 monclient: _check_auth_tickets 2024-03-29T11:23:47.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:47.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f202bc0d640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:23:47.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f202bc0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:47.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 1 -- 172.21.15.67:0/3375453187 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f201c001900 con 0x7f20240ac5a0 2024-03-29T11:23:47.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:47.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:47.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f20289a7640 10 monclient: dump: 2024-03-29T11:23:47.495 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:47.495 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:47.495 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:47.495 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:47.495 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:47.495 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:47.495 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:47.495 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:47.495 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:47.495 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:47.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.512+0000 7f202bc0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:47.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.513+0000 7f20289a7640 1 -- 172.21.15.67:0/3375453187 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f201c001b50 con 0x7f20240ac5a0 2024-03-29T11:23:47.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.513+0000 7f20289a7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:47.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.513+0000 7f20289a7640 1 -- 172.21.15.67:0/3375453187 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f201c001ee0 con 0x7f20240ac5a0 2024-03-29T11:23:47.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.513+0000 7f20289a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:47.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.513+0000 7f20289a7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:47.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.513+0000 7f20289a7640 10 monclient: dump: 2024-03-29T11:23:47.496 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:47.496 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:47.496 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:47.496 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:47.496 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:47.496 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:47.496 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:47.496 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:47.496 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:47.497 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:47.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.513+0000 7f202bc0d640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:47.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.513+0000 7f202bc0d640 4 set_mon_vals no callback set 2024-03-29T11:23:47.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.514+0000 7f202bc0d640 10 monclient: shutdown 2024-03-29T11:23:47.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.514+0000 7f202bc0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:47.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.514+0000 7f202bc0d640 1 -- 172.21.15.67:0/3375453187 >> v1:172.21.15.67:6789/0 conn(0x7f20240ac5a0 legacy=0x7f2024151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:47.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.514+0000 7f202bc0d640 1 -- 172.21.15.67:0/3375453187 shutdown_connections 2024-03-29T11:23:47.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.514+0000 7f202bc0d640 1 -- 172.21.15.67:0/3375453187 >> 172.21.15.67:0/3375453187 conn(0x7f202406b660 msgr2=0x7f202406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:47.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.514+0000 7f202bc0d640 1 -- 172.21.15.67:0/3375453187 shutdown_connections 2024-03-29T11:23:47.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.514+0000 7f202bc0d640 1 -- 172.21.15.67:0/3375453187 wait complete. 2024-03-29T11:23:47.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.515+0000 7f202bc0d640 10 monclient: build_initial_monmap 2024-03-29T11:23:47.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.515+0000 7f202bc0d640 10 monclient: monmap: 2024-03-29T11:23:47.503 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:47.503 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:47.503 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:23:47.504 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:23:47.504 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:47.504 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:47.504 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:47.504 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:23:47.504 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:23:47.504 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:47.504 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.515+0000 7f202bc0d640 1 librados: starting msgr at 2024-03-29T11:23:47.504 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.515+0000 7f202bc0d640 1 librados: starting objecter 2024-03-29T11:23:47.504 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.515+0000 7f202bc0d640 1 Processor -- start 2024-03-29T11:23:47.504 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.516+0000 7f202bc0d640 1 -- start start 2024-03-29T11:23:47.504 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.516+0000 7f202bc0d640 1 librados: setting wanted keys 2024-03-29T11:23:47.504 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.516+0000 7f202bc0d640 1 librados: calling monclient init 2024-03-29T11:23:47.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.516+0000 7f202bc0d640 10 monclient: init 2024-03-29T11:23:47.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.516+0000 7f202bc0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:47.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.516+0000 7f202bc0d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:47.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f202bc0d640 10 monclient(hunting): picked mon.noname-c con 0x7f20240ac5a0 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:47.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f202bc0d640 10 monclient(hunting): picked mon.noname-b con 0x7f20240b0770 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:47.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20299a9640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f20240ac5a0 0x7f20241db580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:35368/0 (socket says 172.21.15.67:35368) 2024-03-29T11:23:47.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20299a9640 1 -- 172.21.15.67:0/3448275054 learned_addr learned my addr 172.21.15.67:0/3448275054 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:47.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f202bc0d640 10 monclient(hunting): picked mon.noname-a con 0x7f202414a700 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:47.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 10 client.?.objecter ms_handle_connect 0x7f20240ac5a0 2024-03-29T11:23:47.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:47.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 10 client.?.objecter ms_handle_connect 0x7f202414a700 2024-03-29T11:23:47.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:47.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f202bc0d640 1 -- 172.21.15.67:0/3448275054 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2024125b40 con 0x7f202414a700 2024-03-29T11:23:47.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f202bc0d640 1 -- 172.21.15.67:0/3448275054 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2024144950 con 0x7f20240b0770 2024-03-29T11:23:47.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f202bc0d640 1 -- 172.21.15.67:0/3448275054 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20241256d0 con 0x7f20240ac5a0 2024-03-29T11:23:47.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f202bc0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:47.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f202bc0d640 10 monclient(hunting): authenticate will time out at 1087.813648s 2024-03-29T11:23:47.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736153178 0 0) 0x7f2024125b40 con 0x7f202414a700 2024-03-29T11:23:47.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:47.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:47.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:47.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:23:47.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2008004eb0 con 0x7f202414a700 2024-03-29T11:23:47.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985327174 0 0) 0x7f20241256d0 con 0x7f20240ac5a0 2024-03-29T11:23:47.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:47.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:47.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:47.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:23:47.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.517+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2024125b40 con 0x7f20240ac5a0 2024-03-29T11:23:47.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 10 client.?.objecter ms_handle_connect 0x7f20240b0770 2024-03-29T11:23:47.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:47.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1926061827 0 0) 0x7f2024125b40 con 0x7f20240ac5a0 2024-03-29T11:23:47.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20241256d0 con 0x7f20240ac5a0 2024-03-29T11:23:47.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f2020001970 con 0x7f20240ac5a0 2024-03-29T11:23:47.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:47.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:47.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 10 monclient(hunting): dump: 2024-03-29T11:23:47.507 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:47.507 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:47.507 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:47.507 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206210434 0 0) 0x7f2024144950 con 0x7f20240b0770 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.518+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2024125b40 con 0x7f20240b0770 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.519+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3414561877 0 0) 0x7f2008004eb0 con 0x7f202414a700 2024-03-29T11:23:47.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.519+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2024144950 con 0x7f202414a700 2024-03-29T11:23:47.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.519+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f201c001e60 con 0x7f202414a700 2024-03-29T11:23:47.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.519+0000 7f20167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:47.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.519+0000 7f20167fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:47.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.519+0000 7f20167fc640 10 monclient(hunting): dump: 2024-03-29T11:23:47.509 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:47.509 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:47.509 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:47.509 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:47.509 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:47.509 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:47.509 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:47.509 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:47.509 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:47.510 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:47.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.519+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2848291215 0 0) 0x7f20241256d0 con 0x7f20240ac5a0 2024-03-29T11:23:47.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.519+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 >> v1:172.21.15.82:6789/0 conn(0x7f20240b0770 legacy=0x7f2024143c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:47.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.519+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 >> v1:172.21.15.67:6789/0 conn(0x7f202414a700 legacy=0x7f20241e6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:47.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.519+0000 7f20167fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:47.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.519+0000 7f20167fc640 1 monclient: found mon.c 2024-03-29T11:23:47.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.519+0000 7f20167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:47.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.519+0000 7f20167fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:47.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.519+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2024125860 con 0x7f20240ac5a0 2024-03-29T11:23:47.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f20167fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:47.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f20167fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:47.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f20167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:47.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f202bc0d640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:23:47.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f202bc0d640 10 monclient: _renew_subs 2024-03-29T11:23:47.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f202bc0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:47.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f202bc0d640 1 -- 172.21.15.67:0/3448275054 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20241dbc90 con 0x7f20240ac5a0 2024-03-29T11:23:47.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f202bc0d640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:47.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f202bc0d640 10 monclient: _renew_subs 2024-03-29T11:23:47.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f202bc0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:47.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f202bc0d640 1 -- 172.21.15.67:0/3448275054 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2024125390 con 0x7f20240ac5a0 2024-03-29T11:23:47.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f202bc0d640 1 librados: init done 2024-03-29T11:23:47.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20200047e0 con 0x7f20240ac5a0 2024-03-29T11:23:47.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f20167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:47.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f2020004c30 con 0x7f20240ac5a0 2024-03-29T11:23:47.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f20167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:47.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f20167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:47.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.520+0000 7f20167fc640 10 monclient: dump: 2024-03-29T11:23:47.511 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:47.512 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:47.512 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:47.512 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:47.512 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:47.512 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:47.512 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:47.512 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:47.512 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:47.512 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:47.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.521+0000 7f2017fff640 4 set_mon_vals no callback set 2024-03-29T11:23:47.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.521+0000 7f202bc0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:47.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.522+0000 7f202bc0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:47.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.522+0000 7f202bc0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:47.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.522+0000 7f202bc0d640 1 -- 172.21.15.67:0/3448275054 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f200c005080 con 0x7f20240ac5a0 2024-03-29T11:23:47.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.525+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f2020004d90 con 0x7f20240ac5a0 2024-03-29T11:23:47.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.525+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f20200849d0 con 0x7f20240ac5a0 2024-03-29T11:23:47.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.526+0000 7f20167fc640 10 client.4406.objecter ms_dispatch 0x7f2024000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:47.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.526+0000 7f20167fc640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:47.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.526+0000 7f20167fc640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:47.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.526+0000 7f20291a8640 10 monclient: get_auth_request con 0x7f200806dd20 auth_method 0 2024-03-29T11:23:47.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.526+0000 7f20167fc640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-29T11:23:47.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.526+0000 7f20167fc640 10 client.4406.objecter ms_handle_connect 0x7f200806dd20 2024-03-29T11:23:47.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.535+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20200502b0 con 0x7f20240ac5a0 2024-03-29T11:23:47.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.535+0000 7f20167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:47.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.535+0000 7f20167fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:47.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.918+0000 7f202bc0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:47.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.918+0000 7f202bc0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:47.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.918+0000 7f202bc0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:23:47.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.918+0000 7f202bc0d640 1 -- 172.21.15.67:0/3448275054 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f200c005330 con 0x7f20240ac5a0 2024-03-29T11:23:47.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.920+0000 7f20167fc640 1 -- 172.21.15.67:0/3448275054 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f20200521e0 con 0x7f20240ac5a0 2024-03-29T11:23:47.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.920+0000 7f20167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:47.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.920+0000 7f20167fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:47.900 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:23:47.900 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:47.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.924+0000 7f202bc0d640 10 librados: watch_flush enter 2024-03-29T11:23:47.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.924+0000 7f202bc0d640 10 librados: watch_flush exit 2024-03-29T11:23:47.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.924+0000 7f202bc0d640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-29T11:23:47.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.924+0000 7f202bc0d640 10 client.4406.objecter successfully canceled tick 2024-03-29T11:23:47.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.924+0000 7f202bc0d640 1 -- 172.21.15.67:0/3448275054 >> v1:172.21.15.67:6816/404787197 conn(0x7f200806dd20 legacy=0x7f20080701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:47.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.924+0000 7f202bc0d640 10 monclient: shutdown 2024-03-29T11:23:47.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.924+0000 7f202bc0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:47.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.924+0000 7f202bc0d640 1 -- 172.21.15.67:0/3448275054 >> v1:172.21.15.67:6790/0 conn(0x7f20240ac5a0 legacy=0x7f20241db580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:47.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.924+0000 7f202bc0d640 1 -- 172.21.15.67:0/3448275054 shutdown_connections 2024-03-29T11:23:47.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.924+0000 7f202bc0d640 1 -- 172.21.15.67:0/3448275054 >> 172.21.15.67:0/3448275054 conn(0x7f202406b660 msgr2=0x7f2024149530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:47.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.924+0000 7f202bc0d640 1 -- 172.21.15.67:0/3448275054 shutdown_connections 2024-03-29T11:23:47.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.924+0000 7f202bc0d640 1 -- 172.21.15.67:0/3448275054 wait complete. 2024-03-29T11:23:47.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:47.925+0000 7f202bc0d640 1 librados: shutdown 2024-03-29T11:23:47.919 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:54.727 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:54.868 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.887+0000 7fbe46845640 10 monclient: get_monmap_and_config 2024-03-29T11:23:54.868 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.887+0000 7fbe46845640 10 monclient: build_initial_monmap 2024-03-29T11:23:54.868 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.887+0000 7fbe46845640 10 monclient: monmap: 2024-03-29T11:23:54.868 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:54.869 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:54.869 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:23:54.889210+0000 2024-03-29T11:23:54.869 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:23:54.889210+0000 2024-03-29T11:23:54.869 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:54.869 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:54.869 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:54.869 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:23:54.869 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:23:54.869 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:54.869 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.888+0000 7fbe46845640 1 Processor -- start 2024-03-29T11:23:54.869 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.888+0000 7fbe46845640 1 -- start start 2024-03-29T11:23:54.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.888+0000 7fbe46845640 10 monclient: init 2024-03-29T11:23:54.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.889+0000 7fbe46845640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:54.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.889+0000 7fbe46845640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:54.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.889+0000 7fbe46845640 10 monclient(hunting): picked mon.noname-c con 0x7fbe401457f0 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:54.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.889+0000 7fbe46845640 10 monclient(hunting): picked mon.noname-a con 0x7fbe4013f650 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:54.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.889+0000 7fbe46845640 10 monclient(hunting): picked mon.noname-b con 0x7fbe400a9d70 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:54.870 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.889+0000 7fbe46845640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe400aaef0 con 0x7fbe4013f650 2024-03-29T11:23:54.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.889+0000 7fbe46845640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe40145bd0 con 0x7fbe401457f0 2024-03-29T11:23:54.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.889+0000 7fbe46845640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe40145eb0 con 0x7fbe400a9d70 2024-03-29T11:23:54.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.889+0000 7fbe46845640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:54.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.889+0000 7fbe46845640 10 monclient(hunting): authenticate will time out at 1095.186461s 2024-03-29T11:23:54.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.889+0000 7fbe3f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbe4013f650 0x7fbe400b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58024/0 (socket says 172.21.15.67:58024) 2024-03-29T11:23:54.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.889+0000 7fbe3f7fe640 1 -- 172.21.15.67:0/265462701 learned_addr learned my addr 172.21.15.67:0/265462701 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:54.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.890+0000 7fbe3effd640 1 -- 172.21.15.67:0/265462701 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770891950 0 0) 0x7fbe40145eb0 con 0x7fbe400a9d70 2024-03-29T11:23:54.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.890+0000 7fbe3effd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:54.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.890+0000 7fbe3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:54.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.890+0000 7fbe3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:54.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.890+0000 7fbe3effd640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:23:54.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.890+0000 7fbe3effd640 1 -- 172.21.15.67:0/265462701 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe2c003520 con 0x7fbe400a9d70 2024-03-29T11:23:54.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.890+0000 7fbe3effd640 1 -- 172.21.15.67:0/265462701 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2694577299 0 0) 0x7fbe2c003520 con 0x7fbe400a9d70 2024-03-29T11:23:54.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.891+0000 7fbe3effd640 1 -- 172.21.15.67:0/265462701 >> v1:172.21.15.82:6789/0 conn(0x7fbe401457f0 legacy=0x7fbe4013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:54.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.892+0000 7fbe3effd640 1 -- 172.21.15.67:0/265462701 >> v1:172.21.15.67:6789/0 conn(0x7fbe4013f650 legacy=0x7fbe400b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:54.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.892+0000 7fbe3effd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:54.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.892+0000 7fbe3effd640 1 monclient: found mon.noname-b 2024-03-29T11:23:54.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.892+0000 7fbe3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:54.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.892+0000 7fbe3effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:23:54.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.892+0000 7fbe3effd640 1 -- 172.21.15.67:0/265462701 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe40146340 con 0x7fbe400a9d70 2024-03-29T11:23:54.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.892+0000 7fbe3effd640 10 monclient: _finish_auth 0 2024-03-29T11:23:54.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.892+0000 7fbe3effd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:54.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.892+0000 7fbe3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:54.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.892+0000 7fbe3effd640 1 -- 172.21.15.67:0/265462701 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fbe30001690 con 0x7fbe400a9d70 2024-03-29T11:23:54.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.892+0000 7fbe3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:54.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.892+0000 7fbe3effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:54.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.892+0000 7fbe3effd640 10 monclient: dump: 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.893+0000 7fbe46845640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.893+0000 7fbe46845640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.893+0000 7fbe3effd640 1 -- 172.21.15.67:0/265462701 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe30001860 con 0x7fbe400a9d70 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.893+0000 7fbe3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:54.876 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.893+0000 7fbe3effd640 1 -- 172.21.15.67:0/265462701 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fbe30001cb0 con 0x7fbe400a9d70 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.893+0000 7fbe46845640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.893+0000 7fbe46845640 4 set_mon_vals no callback set 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.893+0000 7fbe3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.893+0000 7fbe3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.893+0000 7fbe3effd640 10 monclient: dump: 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:54.877 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:54.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.894+0000 7fbe46845640 10 monclient: shutdown 2024-03-29T11:23:54.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.894+0000 7fbe46845640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:54.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.894+0000 7fbe46845640 1 -- 172.21.15.67:0/265462701 >> v1:172.21.15.67:6790/0 conn(0x7fbe400a9d70 legacy=0x7fbe4014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:54.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.894+0000 7fbe46845640 1 -- 172.21.15.67:0/265462701 shutdown_connections 2024-03-29T11:23:54.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.894+0000 7fbe46845640 1 -- 172.21.15.67:0/265462701 >> 172.21.15.67:0/265462701 conn(0x7fbe40066a00 msgr2=0x7fbe40066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:54.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.894+0000 7fbe46845640 1 -- 172.21.15.67:0/265462701 shutdown_connections 2024-03-29T11:23:54.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.894+0000 7fbe46845640 1 -- 172.21.15.67:0/265462701 wait complete. 2024-03-29T11:23:54.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.895+0000 7fbe46845640 10 monclient: build_initial_monmap 2024-03-29T11:23:54.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.895+0000 7fbe46845640 10 monclient: monmap: 2024-03-29T11:23:54.881 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:23:54.881 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:54.881 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:23:54.881 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:23:54.882 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:54.882 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:54.882 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:23:54.882 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:23:54.882 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:23:54.882 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:54.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.896+0000 7fbe46845640 1 librados: starting msgr at 2024-03-29T11:23:54.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.896+0000 7fbe46845640 1 librados: starting objecter 2024-03-29T11:23:54.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.896+0000 7fbe46845640 1 Processor -- start 2024-03-29T11:23:54.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.897+0000 7fbe46845640 1 -- start start 2024-03-29T11:23:54.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.897+0000 7fbe46845640 1 librados: setting wanted keys 2024-03-29T11:23:54.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.897+0000 7fbe46845640 1 librados: calling monclient init 2024-03-29T11:23:54.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.897+0000 7fbe46845640 10 monclient: init 2024-03-29T11:23:54.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.897+0000 7fbe46845640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:54.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.897+0000 7fbe46845640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:54.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.897+0000 7fbe46845640 10 monclient(hunting): picked mon.noname-b con 0x7fbe400a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T11:23:54.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.897+0000 7fbe46845640 10 monclient(hunting): picked mon.noname-c con 0x7fbe4013f650 addr v1:172.21.15.67:6790/0 2024-03-29T11:23:54.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.897+0000 7fbe46845640 10 monclient(hunting): picked mon.noname-a con 0x7fbe401457f0 addr v1:172.21.15.67:6789/0 2024-03-29T11:23:54.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.897+0000 7fbe46845640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe40141b40 con 0x7fbe401457f0 2024-03-29T11:23:54.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.897+0000 7fbe46845640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4006e0a0 con 0x7fbe400a9d70 2024-03-29T11:23:54.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.897+0000 7fbe46845640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe40142aa0 con 0x7fbe4013f650 2024-03-29T11:23:54.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.897+0000 7fbe46845640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:54.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.897+0000 7fbe46845640 10 monclient(hunting): authenticate will time out at 1095.194749s 2024-03-29T11:23:54.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.898+0000 7fbe3f7fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fbe4013f650 0x7fbe4013fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:54030/0 (socket says 172.21.15.67:54030) 2024-03-29T11:23:54.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.898+0000 7fbe3f7fe640 1 -- 172.21.15.67:0/351791218 learned_addr learned my addr 172.21.15.67:0/351791218 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:23:54.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.898+0000 7fbe3cff9640 10 client.?.objecter ms_handle_connect 0x7fbe4013f650 2024-03-29T11:23:54.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.898+0000 7fbe3cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:54.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.899+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927415961 0 0) 0x7fbe40142aa0 con 0x7fbe4013f650 2024-03-29T11:23:54.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.899+0000 7fbe3cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:54.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.899+0000 7fbe3cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:54.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.899+0000 7fbe3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:54.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.899+0000 7fbe3cff9640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:23:54.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.899+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe20004eb0 con 0x7fbe4013f650 2024-03-29T11:23:54.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 10 client.?.objecter ms_handle_connect 0x7fbe401457f0 2024-03-29T11:23:54.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:54.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 10 client.?.objecter ms_handle_connect 0x7fbe400a9d70 2024-03-29T11:23:54.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:54.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215951161 0 0) 0x7fbe40141b40 con 0x7fbe401457f0 2024-03-29T11:23:54.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:54.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:54.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:54.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:23:54.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe40142aa0 con 0x7fbe401457f0 2024-03-29T11:23:54.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458455268 0 0) 0x7fbe4006e0a0 con 0x7fbe400a9d70 2024-03-29T11:23:54.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:54.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:54.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:54.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:23:54.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.900+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe40141b40 con 0x7fbe400a9d70 2024-03-29T11:23:54.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2225703493 0 0) 0x7fbe20004eb0 con 0x7fbe4013f650 2024-03-29T11:23:54.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe4006e0a0 con 0x7fbe4013f650 2024-03-29T11:23:54.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fbe28001be0 con 0x7fbe4013f650 2024-03-29T11:23:54.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:54.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:54.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 10 monclient(hunting): dump: 2024-03-29T11:23:54.890 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:54.890 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:54.890 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:54.890 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:54.890 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:54.890 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2512135957 0 0) 0x7fbe40141b40 con 0x7fbe400a9d70 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe20004eb0 con 0x7fbe400a9d70 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2552483340 0 0) 0x7fbe40142aa0 con 0x7fbe401457f0 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe40141b40 con 0x7fbe401457f0 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fbe380015c0 con 0x7fbe400a9d70 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 10 monclient(hunting): dump: 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:54.891 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fbe30001b70 con 0x7fbe401457f0 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 10 monclient(hunting): dump: 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:54.892 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:54.893 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:54.893 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:54.893 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:54.893 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:54.893 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:54.893 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:54.893 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:54.893 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1536672450 0 0) 0x7fbe40141b40 con 0x7fbe401457f0 2024-03-29T11:23:54.893 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 >> v1:172.21.15.67:6790/0 conn(0x7fbe4013f650 legacy=0x7fbe4013fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:54.893 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 >> v1:172.21.15.82:6789/0 conn(0x7fbe400a9d70 legacy=0x7fbe401df400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:54.893 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:23:54.893 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 1 monclient: found mon.a 2024-03-29T11:23:54.893 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:54.893 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:23:54.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe40142c30 con 0x7fbe401457f0 2024-03-29T11:23:54.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 10 monclient: _finish_auth 0 2024-03-29T11:23:54.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:23:54.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.901+0000 7fbe3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:54.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe46845640 5 monclient: authenticate success, global_id 4386 2024-03-29T11:23:54.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe46845640 10 monclient: _renew_subs 2024-03-29T11:23:54.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe46845640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:23:54.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe46845640 1 -- 172.21.15.67:0/351791218 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe40142450 con 0x7fbe401457f0 2024-03-29T11:23:54.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe46845640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:54.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe46845640 10 monclient: _renew_subs 2024-03-29T11:23:54.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe46845640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:23:54.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe46845640 1 -- 172.21.15.67:0/351791218 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe401dedc0 con 0x7fbe401457f0 2024-03-29T11:23:54.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe46845640 1 librados: init done 2024-03-29T11:23:54.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe30001cd0 con 0x7fbe401457f0 2024-03-29T11:23:54.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe3cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:54.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fbe30003ae0 con 0x7fbe401457f0 2024-03-29T11:23:54.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:54.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe3cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:54.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe3cff9640 10 monclient: dump: 2024-03-29T11:23:54.895 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:23:54.895 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:23:54.895 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:54.895 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:23:54.895 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:54.895 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:23:54.895 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:23:54.895 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:23:54.895 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:23:54.896 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:23:54.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.902+0000 7fbe3e7fc640 4 set_mon_vals no callback set 2024-03-29T11:23:54.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.903+0000 7fbe46845640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:54.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.903+0000 7fbe46845640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:54.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.903+0000 7fbe46845640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:23:54.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.903+0000 7fbe46845640 1 -- 172.21.15.67:0/351791218 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe24005080 con 0x7fbe401457f0 2024-03-29T11:23:54.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.904+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fbe30019ab0 con 0x7fbe401457f0 2024-03-29T11:23:54.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.905+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fbe300837a0 con 0x7fbe401457f0 2024-03-29T11:23:54.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.905+0000 7fbe3cff9640 10 client.4386.objecter ms_dispatch 0x7fbe40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:54.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.905+0000 7fbe3cff9640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:54.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.905+0000 7fbe3cff9640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:54.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.905+0000 7fbe3cff9640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-29T11:23:54.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.906+0000 7fbe3ffff640 10 monclient: get_auth_request con 0x7fbe2006e340 auth_method 0 2024-03-29T11:23:54.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.907+0000 7fbe3cff9640 10 client.4386.objecter ms_handle_connect 0x7fbe2006e340 2024-03-29T11:23:54.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.913+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe3004f100 con 0x7fbe401457f0 2024-03-29T11:23:54.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.913+0000 7fbe3cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:54.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:54.913+0000 7fbe3cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:55.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.233+0000 7fbe46845640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:55.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.233+0000 7fbe46845640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:55.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.233+0000 7fbe46845640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:23:55.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.233+0000 7fbe46845640 1 -- 172.21.15.67:0/351791218 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe24005330 con 0x7fbe401457f0 2024-03-29T11:23:55.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.233+0000 7fbe3cff9640 1 -- 172.21.15.67:0/351791218 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fbe30051030 con 0x7fbe401457f0 2024-03-29T11:23:55.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.233+0000 7fbe3cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:55.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.233+0000 7fbe3cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:55.215 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:23:55.215 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:55.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.237+0000 7fbe46845640 10 librados: watch_flush enter 2024-03-29T11:23:55.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.237+0000 7fbe46845640 10 librados: watch_flush exit 2024-03-29T11:23:55.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.237+0000 7fbe46845640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-29T11:23:55.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.237+0000 7fbe46845640 10 client.4386.objecter successfully canceled tick 2024-03-29T11:23:55.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.238+0000 7fbe46845640 1 -- 172.21.15.67:0/351791218 >> v1:172.21.15.67:6816/404787197 conn(0x7fbe2006e340 legacy=0x7fbe20070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.238+0000 7fbe46845640 10 monclient: shutdown 2024-03-29T11:23:55.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.238+0000 7fbe46845640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:55.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.238+0000 7fbe46845640 1 -- 172.21.15.67:0/351791218 >> v1:172.21.15.67:6789/0 conn(0x7fbe401457f0 legacy=0x7fbe401404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.240+0000 7fbe46845640 1 -- 172.21.15.67:0/351791218 shutdown_connections 2024-03-29T11:23:55.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.240+0000 7fbe46845640 1 -- 172.21.15.67:0/351791218 >> 172.21.15.67:0/351791218 conn(0x7fbe40066a00 msgr2=0x7fbe40066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:55.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.240+0000 7fbe46845640 1 -- 172.21.15.67:0/351791218 shutdown_connections 2024-03-29T11:23:55.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.240+0000 7fbe46845640 1 -- 172.21.15.67:0/351791218 wait complete. 2024-03-29T11:23:55.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:23:55.240+0000 7fbe46845640 1 librados: shutdown 2024-03-29T11:23:55.239 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:02.046 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:02.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.218+0000 7f26c81e9640 10 monclient: get_monmap_and_config 2024-03-29T11:24:02.200 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.218+0000 7f26c81e9640 10 monclient: build_initial_monmap 2024-03-29T11:24:02.200 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.218+0000 7f26c81e9640 10 monclient: monmap: 2024-03-29T11:24:02.200 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:02.200 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:02.200 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:24:02.220121+0000 2024-03-29T11:24:02.200 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:24:02.220121+0000 2024-03-29T11:24:02.200 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:02.200 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:02.200 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:02.200 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:24:02.200 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:24:02.201 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:02.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.219+0000 7f26c81e9640 1 Processor -- start 2024-03-29T11:24:02.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.219+0000 7f26c81e9640 1 -- start start 2024-03-29T11:24:02.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.219+0000 7f26c81e9640 10 monclient: init 2024-03-29T11:24:02.203 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.220+0000 7f26c81e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:02.203 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.220+0000 7f26c81e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:02.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.220+0000 7f26c81e9640 10 monclient(hunting): picked mon.noname-b con 0x7f26c01422b0 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:02.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.221+0000 7f26c81e9640 10 monclient(hunting): picked mon.noname-c con 0x7f26c0142690 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:02.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.221+0000 7f26c81e9640 10 monclient(hunting): picked mon.noname-a con 0x7f26c00b17d0 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:02.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.221+0000 7f26c5f85640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f26c01422b0 0x7f26c013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:56474/0 (socket says 172.21.15.67:56474) 2024-03-29T11:24:02.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.221+0000 7f26c5f85640 1 -- 172.21.15.67:0/3955615205 learned_addr learned my addr 172.21.15.67:0/3955615205 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:02.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.221+0000 7f26c81e9640 1 -- 172.21.15.67:0/3955615205 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c0142c80 con 0x7f26c00b17d0 2024-03-29T11:24:02.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.221+0000 7f26c81e9640 1 -- 172.21.15.67:0/3955615205 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c00aae40 con 0x7f26c0142690 2024-03-29T11:24:02.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.221+0000 7f26c81e9640 1 -- 172.21.15.67:0/3955615205 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c013fbd0 con 0x7f26c01422b0 2024-03-29T11:24:02.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.221+0000 7f26c81e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:02.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.221+0000 7f26c81e9640 10 monclient(hunting): authenticate will time out at 1102.518330s 2024-03-29T11:24:02.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 1 -- 172.21.15.67:0/3955615205 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354939187 0 0) 0x7f26c013fbd0 con 0x7f26c01422b0 2024-03-29T11:24:02.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:24:02.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 1 -- 172.21.15.67:0/3955615205 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ac003520 con 0x7f26c01422b0 2024-03-29T11:24:02.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 1 -- 172.21.15.67:0/3955615205 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848219783 0 0) 0x7f26c00aae40 con 0x7f26c0142690 2024-03-29T11:24:02.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:24:02.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 1 -- 172.21.15.67:0/3955615205 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c013fbd0 con 0x7f26c0142690 2024-03-29T11:24:02.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 1 -- 172.21.15.67:0/3955615205 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647592379 0 0) 0x7f26c0142c80 con 0x7f26c00b17d0 2024-03-29T11:24:02.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:24:02.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.222+0000 7f26c4f83640 1 -- 172.21.15.67:0/3955615205 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c00aae40 con 0x7f26c00b17d0 2024-03-29T11:24:02.208 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 1 -- 172.21.15.67:0/3955615205 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4159550740 0 0) 0x7f26ac003520 con 0x7f26c01422b0 2024-03-29T11:24:02.208 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 1 -- 172.21.15.67:0/3955615205 >> v1:172.21.15.82:6789/0 conn(0x7f26c0142690 legacy=0x7f26c00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.208 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 1 -- 172.21.15.67:0/3955615205 >> v1:172.21.15.67:6789/0 conn(0x7f26c00b17d0 legacy=0x7f26c00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.208 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 10 monclient: _finish_hunting 0 2024-03-29T11:24:02.208 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 1 monclient: found mon.noname-b 2024-03-29T11:24:02.208 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:02.209 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:24:02.209 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 1 -- 172.21.15.67:0/3955615205 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c0151940 con 0x7f26c01422b0 2024-03-29T11:24:02.209 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 10 monclient: _finish_auth 0 2024-03-29T11:24:02.209 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 10 monclient: _check_auth_tickets 2024-03-29T11:24:02.209 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:02.209 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 1 -- 172.21.15.67:0/3955615205 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f26b0001690 con 0x7f26c01422b0 2024-03-29T11:24:02.209 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.209 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:02.209 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c4f83640 10 monclient: dump: 2024-03-29T11:24:02.209 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:02.209 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:02.209 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:02.209 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:02.210 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.210 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:02.210 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:02.210 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:02.210 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:02.210 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:02.210 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c81e9640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:24:02.210 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.223+0000 7f26c81e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:02.210 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.224+0000 7f26c4f83640 1 -- 172.21.15.67:0/3955615205 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26b00018b0 con 0x7f26c01422b0 2024-03-29T11:24:02.210 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.224+0000 7f26c4f83640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:02.210 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.224+0000 7f26c4f83640 1 -- 172.21.15.67:0/3955615205 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f26b0001bc0 con 0x7f26c01422b0 2024-03-29T11:24:02.210 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.224+0000 7f26c81e9640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:02.210 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.224+0000 7f26c81e9640 4 set_mon_vals no callback set 2024-03-29T11:24:02.210 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.225+0000 7f26c4f83640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.225+0000 7f26c4f83640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:02.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.225+0000 7f26c4f83640 10 monclient: dump: 2024-03-29T11:24:02.211 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:02.211 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:02.211 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:02.211 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:02.211 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.211 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:02.211 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:02.211 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:02.211 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:02.211 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:02.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.225+0000 7f26c81e9640 10 monclient: shutdown 2024-03-29T11:24:02.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.225+0000 7f26c81e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:02.212 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.225+0000 7f26c81e9640 1 -- 172.21.15.67:0/3955615205 >> v1:172.21.15.67:6790/0 conn(0x7f26c01422b0 legacy=0x7f26c013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.212 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.225+0000 7f26c81e9640 1 -- 172.21.15.67:0/3955615205 shutdown_connections 2024-03-29T11:24:02.212 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.225+0000 7f26c81e9640 1 -- 172.21.15.67:0/3955615205 >> 172.21.15.67:0/3955615205 conn(0x7f26c0066a00 msgr2=0x7f26c0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:02.212 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.226+0000 7f26c81e9640 1 -- 172.21.15.67:0/3955615205 shutdown_connections 2024-03-29T11:24:02.212 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.226+0000 7f26c81e9640 1 -- 172.21.15.67:0/3955615205 wait complete. 2024-03-29T11:24:02.218 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.227+0000 7f26c81e9640 10 monclient: build_initial_monmap 2024-03-29T11:24:02.218 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.227+0000 7f26c81e9640 10 monclient: monmap: 2024-03-29T11:24:02.218 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:02.218 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:02.218 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:24:02.218 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:24:02.218 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:02.218 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:02.218 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:02.218 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:24:02.218 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:24:02.219 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:02.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.229+0000 7f26c81e9640 1 librados: starting msgr at 2024-03-29T11:24:02.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.229+0000 7f26c81e9640 1 librados: starting objecter 2024-03-29T11:24:02.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.229+0000 7f26c81e9640 1 Processor -- start 2024-03-29T11:24:02.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.230+0000 7f26c81e9640 1 -- start start 2024-03-29T11:24:02.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.230+0000 7f26c81e9640 1 librados: setting wanted keys 2024-03-29T11:24:02.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.230+0000 7f26c81e9640 1 librados: calling monclient init 2024-03-29T11:24:02.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.230+0000 7f26c81e9640 10 monclient: init 2024-03-29T11:24:02.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.230+0000 7f26c81e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:02.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26c81e9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:02.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26c81e9640 10 monclient(hunting): picked mon.noname-a con 0x7f26c00b17d0 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:02.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26c81e9640 10 monclient(hunting): picked mon.noname-b con 0x7f26c01422b0 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:02.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26c5f85640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f26c00b17d0 0x7f26c00b0060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41570/0 (socket says 172.21.15.67:41570) 2024-03-29T11:24:02.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26c5f85640 1 -- 172.21.15.67:0/3025277974 learned_addr learned my addr 172.21.15.67:0/3025277974 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:02.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26c81e9640 10 monclient(hunting): picked mon.noname-c con 0x7f26c0142690 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:02.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26c81e9640 1 -- 172.21.15.67:0/3025277974 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c00b1250 con 0x7f26c00b17d0 2024-03-29T11:24:02.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26b67fc640 10 client.?.objecter ms_handle_connect 0x7f26c00b17d0 2024-03-29T11:24:02.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:02.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26c81e9640 1 -- 172.21.15.67:0/3025277974 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c0143490 con 0x7f26c01422b0 2024-03-29T11:24:02.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26c81e9640 1 -- 172.21.15.67:0/3025277974 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c00afa70 con 0x7f26c0142690 2024-03-29T11:24:02.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26c81e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:02.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26c81e9640 10 monclient(hunting): authenticate will time out at 1102.528306s 2024-03-29T11:24:02.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274367109 0 0) 0x7f26c00b1250 con 0x7f26c00b17d0 2024-03-29T11:24:02.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26b67fc640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:24:02.220 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.231+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a4004eb0 con 0x7f26c00b17d0 2024-03-29T11:24:02.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 client.?.objecter ms_handle_connect 0x7f26c01422b0 2024-03-29T11:24:02.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:02.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 client.?.objecter ms_handle_connect 0x7f26c0142690 2024-03-29T11:24:02.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:02.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2142234014 0 0) 0x7f26a4004eb0 con 0x7f26c00b17d0 2024-03-29T11:24:02.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26c00b1250 con 0x7f26c00b17d0 2024-03-29T11:24:02.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778981799 0 0) 0x7f26c0143490 con 0x7f26c01422b0 2024-03-29T11:24:02.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:24:02.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a4004eb0 con 0x7f26c01422b0 2024-03-29T11:24:02.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807752688 0 0) 0x7f26c00afa70 con 0x7f26c0142690 2024-03-29T11:24:02.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c0143490 con 0x7f26c0142690 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f26b0001650 con 0x7f26c00b17d0 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 monclient(hunting): dump: 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:02.222 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:02.223 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:02.223 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:02.223 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:02.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.232+0000 7f26b67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:02.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 975881123 0 0) 0x7f26c00b1250 con 0x7f26c00b17d0 2024-03-29T11:24:02.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 >> v1:172.21.15.67:6790/0 conn(0x7f26c0142690 legacy=0x7f26c01dade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 >> v1:172.21.15.82:6789/0 conn(0x7f26c01422b0 legacy=0x7f26c014dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:02.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 1 monclient: found mon.a 2024-03-29T11:24:02.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:02.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:24:02.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c00b0770 con 0x7f26c00b17d0 2024-03-29T11:24:02.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:02.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26b0004a90 con 0x7f26c00b17d0 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b7fff640 4 set_mon_vals no callback set 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f26b0004de0 con 0x7f26c00b17d0 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26b67fc640 10 monclient: dump: 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:02.224 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:02.225 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:02.225 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:02.225 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:02.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26c81e9640 5 monclient: authenticate success, global_id 4392 2024-03-29T11:24:02.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26c81e9640 10 monclient: _renew_subs 2024-03-29T11:24:02.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26c81e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:24:02.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.233+0000 7f26c81e9640 1 -- 172.21.15.67:0/3025277974 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c00af5f0 con 0x7f26c00b17d0 2024-03-29T11:24:02.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.234+0000 7f26c81e9640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:02.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.234+0000 7f26c81e9640 10 monclient: _renew_subs 2024-03-29T11:24:02.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.234+0000 7f26c81e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:24:02.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.234+0000 7f26c81e9640 1 -- 172.21.15.67:0/3025277974 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c014e570 con 0x7f26c00b17d0 2024-03-29T11:24:02.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.234+0000 7f26c81e9640 1 librados: init done 2024-03-29T11:24:02.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.236+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f26b001ae30 con 0x7f26c00b17d0 2024-03-29T11:24:02.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.236+0000 7f26c81e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:02.226 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.237+0000 7f26c5784640 10 monclient: get_auth_request con 0x7f26a406dc80 auth_method 0 2024-03-29T11:24:02.226 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.237+0000 7f26c81e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:02.226 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.237+0000 7f26c81e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:24:02.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.237+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f26b0084be0 con 0x7f26c00b17d0 2024-03-29T11:24:02.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.237+0000 7f26b67fc640 10 client.4392.objecter ms_dispatch 0x7f26c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:02.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.237+0000 7f26b67fc640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:02.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.237+0000 7f26b67fc640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:02.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.237+0000 7f26b67fc640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-29T11:24:02.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.237+0000 7f26c81e9640 1 -- 172.21.15.67:0/3025277974 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c0075f10 con 0x7f26c00b17d0 2024-03-29T11:24:02.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.237+0000 7f26b67fc640 10 client.4392.objecter ms_handle_connect 0x7f26a406dc80 2024-03-29T11:24:02.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.245+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26b0050520 con 0x7f26c00b17d0 2024-03-29T11:24:02.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.246+0000 7f26b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:02.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.246+0000 7f26b67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:02.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.632+0000 7f26c81e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:02.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.632+0000 7f26c81e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:02.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.632+0000 7f26c81e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:24:02.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.632+0000 7f26c81e9640 1 -- 172.21.15.67:0/3025277974 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26c00b2270 con 0x7f26c00b17d0 2024-03-29T11:24:02.616 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.634+0000 7f26b67fc640 1 -- 172.21.15.67:0/3025277974 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f26b0084290 con 0x7f26c00b17d0 2024-03-29T11:24:02.616 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.634+0000 7f26b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:02.616 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.634+0000 7f26b67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:02.616 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:24:02.616 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:02.619 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.638+0000 7f26c81e9640 10 librados: watch_flush enter 2024-03-29T11:24:02.619 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.638+0000 7f26c81e9640 10 librados: watch_flush exit 2024-03-29T11:24:02.620 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.638+0000 7f26c81e9640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-29T11:24:02.620 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.638+0000 7f26c81e9640 10 client.4392.objecter successfully canceled tick 2024-03-29T11:24:02.620 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.638+0000 7f26c81e9640 1 -- 172.21.15.67:0/3025277974 >> v1:172.21.15.67:6816/404787197 conn(0x7f26a406dc80 legacy=0x7f26a4070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.620 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.638+0000 7f26c81e9640 10 monclient: shutdown 2024-03-29T11:24:02.620 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.638+0000 7f26c81e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:02.620 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.638+0000 7f26c81e9640 1 -- 172.21.15.67:0/3025277974 >> v1:172.21.15.67:6789/0 conn(0x7f26c00b17d0 legacy=0x7f26c00b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.621 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.638+0000 7f26c81e9640 1 -- 172.21.15.67:0/3025277974 shutdown_connections 2024-03-29T11:24:02.621 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.639+0000 7f26c81e9640 1 -- 172.21.15.67:0/3025277974 >> 172.21.15.67:0/3025277974 conn(0x7f26c0066a00 msgr2=0x7f26c013ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:02.621 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.639+0000 7f26c81e9640 1 -- 172.21.15.67:0/3025277974 shutdown_connections 2024-03-29T11:24:02.621 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.639+0000 7f26c81e9640 1 -- 172.21.15.67:0/3025277974 wait complete. 2024-03-29T11:24:02.621 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:02.639+0000 7f26c81e9640 1 librados: shutdown 2024-03-29T11:24:02.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:09.442 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:09.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.597+0000 7fc00cdfb640 10 monclient: get_monmap_and_config 2024-03-29T11:24:09.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.597+0000 7fc00cdfb640 10 monclient: build_initial_monmap 2024-03-29T11:24:09.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.597+0000 7fc00cdfb640 10 monclient: monmap: 2024-03-29T11:24:09.578 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:09.578 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:09.578 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:24:09.598290+0000 2024-03-29T11:24:09.578 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:24:09.598290+0000 2024-03-29T11:24:09.579 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:09.579 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:09.579 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:09.579 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:24:09.579 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:24:09.579 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:09.581 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.598+0000 7fc00cdfb640 1 Processor -- start 2024-03-29T11:24:09.581 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.598+0000 7fc00cdfb640 1 -- start start 2024-03-29T11:24:09.581 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.598+0000 7fc00cdfb640 10 monclient: init 2024-03-29T11:24:09.581 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.599+0000 7fc00cdfb640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:09.581 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.599+0000 7fc00cdfb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:09.581 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.599+0000 7fc00cdfb640 10 monclient(hunting): picked mon.noname-b con 0x7fc0081457f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:09.581 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.599+0000 7fc00cdfb640 10 monclient(hunting): picked mon.noname-a con 0x7fc00813f650 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:09.581 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.599+0000 7fc00cdfb640 10 monclient(hunting): picked mon.noname-c con 0x7fc0080a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:09.581 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.599+0000 7fc00cdfb640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0080aaef0 con 0x7fc00813f650 2024-03-29T11:24:09.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.599+0000 7fc00cdfb640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc008145bd0 con 0x7fc0080a9d70 2024-03-29T11:24:09.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.599+0000 7fc00cdfb640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc008145eb0 con 0x7fc0081457f0 2024-03-29T11:24:09.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.599+0000 7fc00cdfb640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:09.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.599+0000 7fc00cdfb640 10 monclient(hunting): authenticate will time out at 1109.895494s 2024-03-29T11:24:09.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.599+0000 7fc00659c640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fc0081457f0 0x7fc00813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:42520/0 (socket says 172.21.15.67:42520) 2024-03-29T11:24:09.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.599+0000 7fc00659c640 1 -- 172.21.15.67:0/4125488916 learned_addr learned my addr 172.21.15.67:0/4125488916 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:09.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.600+0000 7fc00559a640 1 -- 172.21.15.67:0/4125488916 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468223772 0 0) 0x7fc008145eb0 con 0x7fc0081457f0 2024-03-29T11:24:09.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.600+0000 7fc00559a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:09.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.600+0000 7fc00559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:09.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.600+0000 7fc00559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:09.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.600+0000 7fc00559a640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:24:09.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.600+0000 7fc00559a640 1 -- 172.21.15.67:0/4125488916 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbff0003520 con 0x7fc0081457f0 2024-03-29T11:24:09.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.600+0000 7fc00559a640 1 -- 172.21.15.67:0/4125488916 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2320397264 0 0) 0x7fbff0003520 con 0x7fc0081457f0 2024-03-29T11:24:09.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.600+0000 7fc00559a640 1 -- 172.21.15.67:0/4125488916 >> v1:172.21.15.82:6789/0 conn(0x7fc0080a9d70 legacy=0x7fc00814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:09.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.600+0000 7fc00559a640 1 -- 172.21.15.67:0/4125488916 >> v1:172.21.15.67:6789/0 conn(0x7fc00813f650 legacy=0x7fc0080b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:09.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:09.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 1 monclient: found mon.noname-b 2024-03-29T11:24:09.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:09.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:24:09.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 1 -- 172.21.15.67:0/4125488916 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc008146340 con 0x7fc0081457f0 2024-03-29T11:24:09.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 10 monclient: _finish_auth 0 2024-03-29T11:24:09.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 1 -- 172.21.15.67:0/4125488916 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fbff8001690 con 0x7fc0081457f0 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 10 monclient: dump: 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:09.585 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:09.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00cdfb640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:24:09.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00cdfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:09.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 1 -- 172.21.15.67:0/4125488916 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbff8001920 con 0x7fc0081457f0 2024-03-29T11:24:09.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:09.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 1 -- 172.21.15.67:0/4125488916 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fbff8001cb0 con 0x7fc0081457f0 2024-03-29T11:24:09.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:09.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:09.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00559a640 10 monclient: dump: 2024-03-29T11:24:09.586 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:09.586 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:09.586 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:09.586 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:09.586 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:09.586 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:09.587 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:09.587 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:09.587 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:09.587 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:09.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00cdfb640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:09.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.602+0000 7fc00cdfb640 4 set_mon_vals no callback set 2024-03-29T11:24:09.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.603+0000 7fc00cdfb640 10 monclient: shutdown 2024-03-29T11:24:09.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.603+0000 7fc00cdfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:09.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.603+0000 7fc00cdfb640 1 -- 172.21.15.67:0/4125488916 >> v1:172.21.15.67:6790/0 conn(0x7fc0081457f0 legacy=0x7fc00813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:09.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.603+0000 7fc00cdfb640 1 -- 172.21.15.67:0/4125488916 shutdown_connections 2024-03-29T11:24:09.593 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.603+0000 7fc00cdfb640 1 -- 172.21.15.67:0/4125488916 >> 172.21.15.67:0/4125488916 conn(0x7fc008066a00 msgr2=0x7fc008066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:09.593 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.603+0000 7fc00cdfb640 1 -- 172.21.15.67:0/4125488916 shutdown_connections 2024-03-29T11:24:09.593 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.604+0000 7fc00cdfb640 1 -- 172.21.15.67:0/4125488916 wait complete. 2024-03-29T11:24:09.593 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.605+0000 7fc00cdfb640 10 monclient: build_initial_monmap 2024-03-29T11:24:09.593 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.605+0000 7fc00cdfb640 10 monclient: monmap: 2024-03-29T11:24:09.593 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:09.593 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:09.593 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:24:09.593 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:24:09.593 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:09.593 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:09.593 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:09.594 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:24:09.594 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:24:09.594 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:09.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.606+0000 7fc00cdfb640 1 librados: starting msgr at 2024-03-29T11:24:09.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.606+0000 7fc00cdfb640 1 librados: starting objecter 2024-03-29T11:24:09.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.606+0000 7fc00cdfb640 1 Processor -- start 2024-03-29T11:24:09.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.606+0000 7fc00cdfb640 1 -- start start 2024-03-29T11:24:09.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.606+0000 7fc00cdfb640 1 librados: setting wanted keys 2024-03-29T11:24:09.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.606+0000 7fc00cdfb640 1 librados: calling monclient init 2024-03-29T11:24:09.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.606+0000 7fc00cdfb640 10 monclient: init 2024-03-29T11:24:09.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.607+0000 7fc00cdfb640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:09.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.607+0000 7fc00cdfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:09.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.607+0000 7fc00cdfb640 10 monclient(hunting): picked mon.noname-a con 0x7fc0080a9d70 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:09.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.607+0000 7fc00cdfb640 10 monclient(hunting): picked mon.noname-b con 0x7fc00813f650 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:09.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.607+0000 7fc00cdfb640 10 monclient(hunting): picked mon.noname-c con 0x7fc0081457f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:09.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.607+0000 7fc00cdfb640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00813a3d0 con 0x7fc0080a9d70 2024-03-29T11:24:09.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.607+0000 7fc00cdfb640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc008139de0 con 0x7fc00813f650 2024-03-29T11:24:09.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.607+0000 7fc00cdfb640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc008139f70 con 0x7fc0081457f0 2024-03-29T11:24:09.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.607+0000 7fc00cdfb640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:09.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.607+0000 7fc00cdfb640 10 monclient(hunting): authenticate will time out at 1109.903892s 2024-03-29T11:24:09.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fc006d9d640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fc0081457f0 0x7fc00813c260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:42922/0 (socket says 172.21.15.67:42922) 2024-03-29T11:24:09.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fc006d9d640 1 -- 172.21.15.67:0/589612739 learned_addr learned my addr 172.21.15.67:0/589612739 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:09.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 10 client.?.objecter ms_handle_connect 0x7fc0081457f0 2024-03-29T11:24:09.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:09.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514396181 0 0) 0x7fc008139f70 con 0x7fc0081457f0 2024-03-29T11:24:09.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:09.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:09.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:09.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:24:09.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfe8004eb0 con 0x7fc0081457f0 2024-03-29T11:24:09.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3407038792 0 0) 0x7fbfe8004eb0 con 0x7fc0081457f0 2024-03-29T11:24:09.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc008139f70 con 0x7fc0081457f0 2024-03-29T11:24:09.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc000001c60 con 0x7fc0081457f0 2024-03-29T11:24:09.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:09.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:09.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:09.596 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:09.596 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:09.597 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:09.597 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:09.597 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:09.597 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:09.597 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:09.597 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:09.597 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:09.597 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:09.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.608+0000 7fbff6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:09.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.609+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1248651912 0 0) 0x7fc008139f70 con 0x7fc0081457f0 2024-03-29T11:24:09.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.609+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 >> v1:172.21.15.82:6789/0 conn(0x7fc00813f650 legacy=0x7fc0081da5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:09.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.609+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 >> v1:172.21.15.67:6789/0 conn(0x7fc0080a9d70 legacy=0x7fc00813a8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:09.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.610+0000 7fbff6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:09.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.610+0000 7fbff6ffd640 1 monclient: found mon.c 2024-03-29T11:24:09.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.610+0000 7fbff6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:09.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.610+0000 7fbff6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:09.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.610+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc008142380 con 0x7fc0081457f0 2024-03-29T11:24:09.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.610+0000 7fbff6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:09.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.610+0000 7fbff6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:09.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.610+0000 7fbff6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:09.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.610+0000 7fc00cdfb640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:24:09.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.611+0000 7fc00cdfb640 10 monclient: _renew_subs 2024-03-29T11:24:09.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.611+0000 7fc00cdfb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:09.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.611+0000 7fc00cdfb640 1 -- 172.21.15.67:0/589612739 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0081425d0 con 0x7fc0081457f0 2024-03-29T11:24:09.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.611+0000 7fc00cdfb640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:09.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.611+0000 7fc00cdfb640 10 monclient: _renew_subs 2024-03-29T11:24:09.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.611+0000 7fc00cdfb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:09.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.611+0000 7fc00cdfb640 1 -- 172.21.15.67:0/589612739 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc008135f20 con 0x7fc0081457f0 2024-03-29T11:24:09.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.611+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0000020b0 con 0x7fc0081457f0 2024-03-29T11:24:09.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.611+0000 7fc00cdfb640 1 librados: init done 2024-03-29T11:24:09.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.611+0000 7fbff6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:09.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.611+0000 7fc004d99640 4 set_mon_vals no callback set 2024-03-29T11:24:09.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.612+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fc000003c20 con 0x7fc0081457f0 2024-03-29T11:24:09.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.612+0000 7fbff6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:09.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.612+0000 7fbff6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:09.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.612+0000 7fbff6ffd640 10 monclient: dump: 2024-03-29T11:24:09.599 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:09.599 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:09.599 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:09.599 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:09.599 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:09.599 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:09.600 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:09.600 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:09.600 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:09.600 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:09.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.613+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fc000003d80 con 0x7fc0081457f0 2024-03-29T11:24:09.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.613+0000 7fc00cdfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:09.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.613+0000 7fc00cdfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:09.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.613+0000 7fc00cdfb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:09.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.614+0000 7fc00cdfb640 1 -- 172.21.15.67:0/589612739 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00813a5b0 con 0x7fc0081457f0 2024-03-29T11:24:09.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.614+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fc000084df0 con 0x7fc0081457f0 2024-03-29T11:24:09.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.614+0000 7fc00659c640 10 monclient: get_auth_request con 0x7fbfe806db30 auth_method 0 2024-03-29T11:24:09.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.614+0000 7fbff6ffd640 10 client.4424.objecter ms_dispatch 0x7fc008000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:09.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.614+0000 7fbff6ffd640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:09.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.614+0000 7fbff6ffd640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:09.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.614+0000 7fbff6ffd640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T11:24:09.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.614+0000 7fbff6ffd640 10 client.4424.objecter ms_handle_connect 0x7fbfe806db30 2024-03-29T11:24:09.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.621+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0000506d0 con 0x7fc0081457f0 2024-03-29T11:24:09.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.621+0000 7fbff6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:09.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.621+0000 7fbff6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:09.925 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.944+0000 7fc00cdfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:09.925 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.944+0000 7fc00cdfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:09.925 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.944+0000 7fc00cdfb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:09.925 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.944+0000 7fc00cdfb640 1 -- 172.21.15.67:0/589612739 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc008075f10 con 0x7fc0081457f0 2024-03-29T11:24:09.925 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:24:09.925 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:09.926 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.944+0000 7fbff6ffd640 1 -- 172.21.15.67:0/589612739 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fc000052600 con 0x7fc0081457f0 2024-03-29T11:24:09.926 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.945+0000 7fbff6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:09.926 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.945+0000 7fbff6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:09.929 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.948+0000 7fc00cdfb640 10 librados: watch_flush enter 2024-03-29T11:24:09.929 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.948+0000 7fc00cdfb640 10 librados: watch_flush exit 2024-03-29T11:24:09.929 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.948+0000 7fc00cdfb640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T11:24:09.929 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.948+0000 7fc00cdfb640 10 client.4424.objecter successfully canceled tick 2024-03-29T11:24:09.929 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.948+0000 7fc00cdfb640 1 -- 172.21.15.67:0/589612739 >> v1:172.21.15.67:6816/404787197 conn(0x7fbfe806db30 legacy=0x7fbfe806fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:09.929 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.948+0000 7fc00cdfb640 10 monclient: shutdown 2024-03-29T11:24:09.929 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.948+0000 7fc00cdfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:09.929 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.948+0000 7fc00cdfb640 1 -- 172.21.15.67:0/589612739 >> v1:172.21.15.67:6790/0 conn(0x7fc0081457f0 legacy=0x7fc00813c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:09.929 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.948+0000 7fc00cdfb640 1 -- 172.21.15.67:0/589612739 shutdown_connections 2024-03-29T11:24:09.930 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.948+0000 7fc00cdfb640 1 -- 172.21.15.67:0/589612739 >> 172.21.15.67:0/589612739 conn(0x7fc008066a00 msgr2=0x7fc0080b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:09.930 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.948+0000 7fc00cdfb640 1 -- 172.21.15.67:0/589612739 shutdown_connections 2024-03-29T11:24:09.930 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.948+0000 7fc00cdfb640 1 -- 172.21.15.67:0/589612739 wait complete. 2024-03-29T11:24:09.930 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:09.948+0000 7fc00cdfb640 1 librados: shutdown 2024-03-29T11:24:09.944 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:16.750 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:16.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.902+0000 7f0c6d219640 10 monclient: get_monmap_and_config 2024-03-29T11:24:16.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.902+0000 7f0c6d219640 10 monclient: build_initial_monmap 2024-03-29T11:24:16.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.902+0000 7f0c6d219640 10 monclient: monmap: 2024-03-29T11:24:16.884 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:16.884 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:16.884 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:24:16.903560+0000 2024-03-29T11:24:16.884 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:24:16.903560+0000 2024-03-29T11:24:16.884 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:16.884 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:16.884 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:16.884 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:24:16.884 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:24:16.884 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:16.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.903+0000 7f0c6d219640 1 Processor -- start 2024-03-29T11:24:16.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.903+0000 7f0c6d219640 1 -- start start 2024-03-29T11:24:16.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.903+0000 7f0c6d219640 10 monclient: init 2024-03-29T11:24:16.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.904+0000 7f0c6d219640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:16.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.904+0000 7f0c6d219640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:16.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.904+0000 7f0c6d219640 10 monclient(hunting): picked mon.noname-a con 0x7f0c600389e0 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:16.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.904+0000 7f0c6d219640 10 monclient(hunting): picked mon.noname-c con 0x7f0c60038dc0 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:16.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.904+0000 7f0c6d219640 10 monclient(hunting): picked mon.noname-b con 0x7f0c60030660 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:16.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.904+0000 7f0c6d219640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c600c54c0 con 0x7f0c600389e0 2024-03-29T11:24:16.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.904+0000 7f0c6ca18640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f0c60030660 0x7f0c600d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:45274/0 (socket says 172.21.15.67:45274) 2024-03-29T11:24:16.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.904+0000 7f0c6ca18640 1 -- 172.21.15.67:0/3750966036 learned_addr learned my addr 172.21.15.67:0/3750966036 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:16.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.904+0000 7f0c6d219640 1 -- 172.21.15.67:0/3750966036 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c60036430 con 0x7f0c60038dc0 2024-03-29T11:24:16.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.904+0000 7f0c6d219640 1 -- 172.21.15.67:0/3750966036 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c60039840 con 0x7f0c60030660 2024-03-29T11:24:16.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.905+0000 7f0c6d219640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:16.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.905+0000 7f0c6d219640 10 monclient(hunting): authenticate will time out at 1117.200969s 2024-03-29T11:24:16.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.905+0000 7f0c66ffd640 1 -- 172.21.15.67:0/3750966036 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050775850 0 0) 0x7f0c60039840 con 0x7f0c60030660 2024-03-29T11:24:16.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.905+0000 7f0c66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:16.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.905+0000 7f0c66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:16.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.905+0000 7f0c66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:16.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.905+0000 7f0c66ffd640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:24:16.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.905+0000 7f0c66ffd640 1 -- 172.21.15.67:0/3750966036 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c58003520 con 0x7f0c60030660 2024-03-29T11:24:16.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.905+0000 7f0c66ffd640 1 -- 172.21.15.67:0/3750966036 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2453332814 0 0) 0x7f0c58003520 con 0x7f0c60030660 2024-03-29T11:24:16.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.905+0000 7f0c66ffd640 1 -- 172.21.15.67:0/3750966036 >> v1:172.21.15.82:6789/0 conn(0x7f0c60038dc0 legacy=0x7f0c600cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:16.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.905+0000 7f0c66ffd640 1 -- 172.21.15.67:0/3750966036 >> v1:172.21.15.67:6789/0 conn(0x7f0c600389e0 legacy=0x7f0c60033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:16.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.907+0000 7f0c66ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:16.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.907+0000 7f0c66ffd640 1 monclient: found mon.noname-b 2024-03-29T11:24:16.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.907+0000 7f0c66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:16.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.907+0000 7f0c66ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:24:16.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.907+0000 7f0c66ffd640 1 -- 172.21.15.67:0/3750966036 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c60035690 con 0x7f0c60030660 2024-03-29T11:24:16.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.907+0000 7f0c66ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:16.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.907+0000 7f0c66ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:16.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.907+0000 7f0c66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:16.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.907+0000 7f0c66ffd640 1 -- 172.21.15.67:0/3750966036 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0c6804f730 con 0x7f0c60030660 2024-03-29T11:24:16.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.907+0000 7f0c66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:16.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.907+0000 7f0c66ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:16.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.907+0000 7f0c66ffd640 10 monclient: dump: 2024-03-29T11:24:16.890 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:16.890 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:16.890 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:16.890 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:16.890 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:16.890 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:16.890 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:16.890 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:16.890 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:16.890 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:16.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.907+0000 7f0c6d219640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:24:16.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.907+0000 7f0c6d219640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:16.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.908+0000 7f0c66ffd640 1 -- 172.21.15.67:0/3750966036 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c680008d0 con 0x7f0c60030660 2024-03-29T11:24:16.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.908+0000 7f0c66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:16.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.908+0000 7f0c66ffd640 1 -- 172.21.15.67:0/3750966036 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0c680758b0 con 0x7f0c60030660 2024-03-29T11:24:16.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.908+0000 7f0c66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:16.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.908+0000 7f0c66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:16.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.908+0000 7f0c66ffd640 10 monclient: dump: 2024-03-29T11:24:16.891 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:16.891 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:16.891 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:16.891 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:16.891 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:16.891 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:16.891 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:16.892 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:16.892 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:16.892 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:16.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.908+0000 7f0c6d219640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:16.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.908+0000 7f0c6d219640 4 set_mon_vals no callback set 2024-03-29T11:24:16.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.908+0000 7f0c6d219640 10 monclient: shutdown 2024-03-29T11:24:16.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.909+0000 7f0c6d219640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:16.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.909+0000 7f0c6d219640 1 -- 172.21.15.67:0/3750966036 >> v1:172.21.15.67:6790/0 conn(0x7f0c60030660 legacy=0x7f0c600d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:16.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.909+0000 7f0c6d219640 1 -- 172.21.15.67:0/3750966036 shutdown_connections 2024-03-29T11:24:16.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.909+0000 7f0c6d219640 1 -- 172.21.15.67:0/3750966036 >> 172.21.15.67:0/3750966036 conn(0x7f0c60005c00 msgr2=0x7f0c60006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:16.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.909+0000 7f0c6d219640 1 -- 172.21.15.67:0/3750966036 shutdown_connections 2024-03-29T11:24:16.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.909+0000 7f0c6d219640 1 -- 172.21.15.67:0/3750966036 wait complete. 2024-03-29T11:24:16.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.910+0000 7f0c6d219640 10 monclient: build_initial_monmap 2024-03-29T11:24:16.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.910+0000 7f0c6d219640 10 monclient: monmap: 2024-03-29T11:24:16.896 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:16.896 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:16.896 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:24:16.896 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:24:16.896 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:16.896 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:16.896 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:16.896 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:24:16.897 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:24:16.897 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:16.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.910+0000 7f0c6d219640 1 librados: starting msgr at 2024-03-29T11:24:16.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.910+0000 7f0c6d219640 1 librados: starting objecter 2024-03-29T11:24:16.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.910+0000 7f0c6d219640 1 Processor -- start 2024-03-29T11:24:16.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.911+0000 7f0c6d219640 1 -- start start 2024-03-29T11:24:16.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.911+0000 7f0c6d219640 1 librados: setting wanted keys 2024-03-29T11:24:16.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.911+0000 7f0c6d219640 1 librados: calling monclient init 2024-03-29T11:24:16.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.911+0000 7f0c6d219640 10 monclient: init 2024-03-29T11:24:16.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.911+0000 7f0c6d219640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:16.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.911+0000 7f0c6d219640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:16.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.912+0000 7f0c6d219640 10 monclient(hunting): picked mon.noname-b con 0x7f0c60030660 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:16.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.912+0000 7f0c6d219640 10 monclient(hunting): picked mon.noname-c con 0x7f0c600389e0 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:16.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.912+0000 7f0c67fff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f0c60030660 0x7f0c6015d1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:33414/0 (socket says 172.21.15.67:33414) 2024-03-29T11:24:16.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.912+0000 7f0c6d219640 10 monclient(hunting): picked mon.noname-a con 0x7f0c6002fca0 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:16.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.912+0000 7f0c677fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f0c600389e0 0x7f0c600c74b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:46024/0 (socket says 172.21.15.67:46024) 2024-03-29T11:24:16.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.913+0000 7f0c677fe640 1 -- 172.21.15.67:0/326385991 learned_addr learned my addr 172.21.15.67:0/326385991 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:16.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.913+0000 7f0c6d219640 1 -- 172.21.15.67:0/326385991 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6015cc80 con 0x7f0c6002fca0 2024-03-29T11:24:16.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.913+0000 7f0c6d219640 1 -- 172.21.15.67:0/326385991 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6002f430 con 0x7f0c60030660 2024-03-29T11:24:16.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.913+0000 7f0c6d219640 1 -- 172.21.15.67:0/326385991 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6015c830 con 0x7f0c600389e0 2024-03-29T11:24:16.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.913+0000 7f0c6d219640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:16.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.913+0000 7f0c6d219640 10 monclient(hunting): authenticate will time out at 1117.209376s 2024-03-29T11:24:16.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.913+0000 7f0c64ff9640 10 client.?.objecter ms_handle_connect 0x7f0c600389e0 2024-03-29T11:24:16.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.913+0000 7f0c64ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:16.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.913+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257442640 0 0) 0x7f0c6015c830 con 0x7f0c600389e0 2024-03-29T11:24:16.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.913+0000 7f0c64ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:16.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.913+0000 7f0c64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:16.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.913+0000 7f0c64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:16.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.913+0000 7f0c64ff9640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:24:16.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.913+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c4c004eb0 con 0x7f0c600389e0 2024-03-29T11:24:16.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 10 client.?.objecter ms_handle_connect 0x7f0c60030660 2024-03-29T11:24:16.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:16.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 294946037 0 0) 0x7f0c4c004eb0 con 0x7f0c600389e0 2024-03-29T11:24:16.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c6015c830 con 0x7f0c600389e0 2024-03-29T11:24:16.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0c54001960 con 0x7f0c600389e0 2024-03-29T11:24:16.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:16.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:16.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 10 monclient(hunting): dump: 2024-03-29T11:24:16.899 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:16.900 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:16.900 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:16.900 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:16.900 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:16.900 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:16.900 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:16.900 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:16.900 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:16.900 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:16.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:16.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688535661 0 0) 0x7f0c6002f430 con 0x7f0c60030660 2024-03-29T11:24:16.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:16.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:16.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:16.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:24:16.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c4c005680 con 0x7f0c60030660 2024-03-29T11:24:16.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 463024217 0 0) 0x7f0c4c005680 con 0x7f0c60030660 2024-03-29T11:24:16.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c6002f430 con 0x7f0c60030660 2024-03-29T11:24:16.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0c5c0017e0 con 0x7f0c60030660 2024-03-29T11:24:16.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:16.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:16.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 10 monclient(hunting): dump: 2024-03-29T11:24:16.901 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:16.901 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:16.901 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:16.901 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:16.901 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:16.901 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:16.902 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:16.902 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:16.902 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:16.902 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:16.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.914+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3427765874 0 0) 0x7f0c6002f430 con 0x7f0c60030660 2024-03-29T11:24:16.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.915+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 >> v1:172.21.15.67:6790/0 conn(0x7f0c600389e0 legacy=0x7f0c600c74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:16.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.915+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 >> v1:172.21.15.67:6789/0 conn(0x7f0c6002fca0 legacy=0x7f0c6002e390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:16.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c64ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:16.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c64ff9640 1 monclient: found mon.b 2024-03-29T11:24:16.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c64ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:16.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c64ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:24:16.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6015c9c0 con 0x7f0c60030660 2024-03-29T11:24:16.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c64ff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:16.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c64ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:16.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:16.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c6d219640 5 monclient: authenticate success, global_id 4372 2024-03-29T11:24:16.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c6d219640 10 monclient: _renew_subs 2024-03-29T11:24:16.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c6d219640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:24:16.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c6d219640 1 -- 172.21.15.67:0/326385991 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6002eaa0 con 0x7f0c60030660 2024-03-29T11:24:16.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c6d219640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:16.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c6d219640 10 monclient: _renew_subs 2024-03-29T11:24:16.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c6d219640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:24:16.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c6d219640 1 -- 172.21.15.67:0/326385991 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c600c5aa0 con 0x7f0c60030660 2024-03-29T11:24:16.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.916+0000 7f0c6d219640 1 librados: init done 2024-03-29T11:24:16.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.917+0000 7f0c6d219640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:16.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.917+0000 7f0c6d219640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:16.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.917+0000 7f0c6d219640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:24:16.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.917+0000 7f0c6d219640 1 -- 172.21.15.67:0/326385991 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c60007b90 con 0x7f0c60030660 2024-03-29T11:24:16.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.918+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c5c001cf0 con 0x7f0c60030660 2024-03-29T11:24:16.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.918+0000 7f0c64ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:16.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.918+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0c5c003790 con 0x7f0c60030660 2024-03-29T11:24:16.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.918+0000 7f0c64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:16.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.918+0000 7f0c64ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:16.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.918+0000 7f0c64ff9640 10 monclient: dump: 2024-03-29T11:24:16.906 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:16.906 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:16.906 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:16.906 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.918+0000 7f0c65ffb640 4 set_mon_vals no callback set 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.920+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f0c5c0038f0 con 0x7f0c60030660 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.920+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f0c5c084830 con 0x7f0c60030660 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.920+0000 7f0c64ff9640 10 client.4372.objecter ms_dispatch 0x7f0c68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.920+0000 7f0c64ff9640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.920+0000 7f0c64ff9640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.920+0000 7f0c64ff9640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.921+0000 7f0c677fe640 10 monclient: get_auth_request con 0x7f0c4c06e180 auth_method 0 2024-03-29T11:24:16.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.921+0000 7f0c64ff9640 10 client.4372.objecter ms_handle_connect 0x7f0c4c06e180 2024-03-29T11:24:16.908 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.926+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c5c050110 con 0x7f0c60030660 2024-03-29T11:24:16.908 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.926+0000 7f0c64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:16.908 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:16.926+0000 7f0c64ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:17.216 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.234+0000 7f0c6d219640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:17.216 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.234+0000 7f0c6d219640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:17.217 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.234+0000 7f0c6d219640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:24:17.217 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.234+0000 7f0c6d219640 1 -- 172.21.15.67:0/326385991 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c60005120 con 0x7f0c60030660 2024-03-29T11:24:17.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.237+0000 7f0c64ff9640 1 -- 172.21.15.67:0/326385991 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f0c5c052040 con 0x7f0c60030660 2024-03-29T11:24:17.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.238+0000 7f0c64ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:17.219 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.238+0000 7f0c64ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:17.219 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:24:17.219 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:17.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.240+0000 7f0c6d219640 10 librados: watch_flush enter 2024-03-29T11:24:17.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.240+0000 7f0c6d219640 10 librados: watch_flush exit 2024-03-29T11:24:17.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.240+0000 7f0c6d219640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-29T11:24:17.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.240+0000 7f0c6d219640 10 client.4372.objecter successfully canceled tick 2024-03-29T11:24:17.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.240+0000 7f0c6d219640 1 -- 172.21.15.67:0/326385991 >> v1:172.21.15.67:6816/404787197 conn(0x7f0c4c06e180 legacy=0x7f0c4c070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.240+0000 7f0c6d219640 10 monclient: shutdown 2024-03-29T11:24:17.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.240+0000 7f0c6d219640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:17.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.240+0000 7f0c6d219640 1 -- 172.21.15.67:0/326385991 >> v1:172.21.15.82:6789/0 conn(0x7f0c60030660 legacy=0x7f0c6015d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.240+0000 7f0c6d219640 1 -- 172.21.15.67:0/326385991 shutdown_connections 2024-03-29T11:24:17.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.240+0000 7f0c6d219640 1 -- 172.21.15.67:0/326385991 >> 172.21.15.67:0/326385991 conn(0x7f0c60005c00 msgr2=0x7f0c60034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:17.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.240+0000 7f0c6d219640 1 -- 172.21.15.67:0/326385991 shutdown_connections 2024-03-29T11:24:17.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.240+0000 7f0c6d219640 1 -- 172.21.15.67:0/326385991 wait complete. 2024-03-29T11:24:17.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:17.241+0000 7f0c6d219640 1 librados: shutdown 2024-03-29T11:24:17.236 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:24.043 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:24.190 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.207+0000 7f708611c640 10 monclient: get_monmap_and_config 2024-03-29T11:24:24.190 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.207+0000 7f708611c640 10 monclient: build_initial_monmap 2024-03-29T11:24:24.190 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.207+0000 7f708611c640 10 monclient: monmap: 2024-03-29T11:24:24.190 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:24.190 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:24.190 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:24:24.208213+0000 2024-03-29T11:24:24.190 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:24:24.208213+0000 2024-03-29T11:24:24.190 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:24.191 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:24.191 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:24.191 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:24:24.191 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:24:24.191 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:24.191 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.208+0000 7f708611c640 1 Processor -- start 2024-03-29T11:24:24.191 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.208+0000 7f708611c640 1 -- start start 2024-03-29T11:24:24.191 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.208+0000 7f708611c640 10 monclient: init 2024-03-29T11:24:24.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.209+0000 7f708611c640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:24.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.209+0000 7f708611c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:24.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.209+0000 7f708611c640 10 monclient(hunting): picked mon.noname-c con 0x7f70800a9930 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:24.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.209+0000 7f708611c640 10 monclient(hunting): picked mon.noname-a con 0x7f7080149dc0 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:24.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.209+0000 7f708611c640 10 monclient(hunting): picked mon.noname-b con 0x7f708014ff40 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:24.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.209+0000 7f708611c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70800aae60 con 0x7f7080149dc0 2024-03-29T11:24:24.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.209+0000 7f708611c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70800b4310 con 0x7f70800a9930 2024-03-29T11:24:24.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.209+0000 7f708611c640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70800afdd0 con 0x7f708014ff40 2024-03-29T11:24:24.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.209+0000 7f708611c640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:24.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.209+0000 7f708611c640 10 monclient(hunting): authenticate will time out at 1124.505402s 2024-03-29T11:24:24.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.211+0000 7f707ffff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f708014ff40 0x7f7080152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:46714/0 (socket says 172.21.15.67:46714) 2024-03-29T11:24:24.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.211+0000 7f707ffff640 1 -- 172.21.15.67:0/981281517 learned_addr learned my addr 172.21.15.67:0/981281517 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:24.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 1 -- 172.21.15.67:0/981281517 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40619699 0 0) 0x7f70800afdd0 con 0x7f708014ff40 2024-03-29T11:24:24.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:24:24.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 1 -- 172.21.15.67:0/981281517 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7068003520 con 0x7f708014ff40 2024-03-29T11:24:24.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 1 -- 172.21.15.67:0/981281517 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272118160 0 0) 0x7f70800b4310 con 0x7f70800a9930 2024-03-29T11:24:24.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:24:24.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 1 -- 172.21.15.67:0/981281517 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70800afdd0 con 0x7f70800a9930 2024-03-29T11:24:24.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 1 -- 172.21.15.67:0/981281517 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135761540 0 0) 0x7f70800aae60 con 0x7f7080149dc0 2024-03-29T11:24:24.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:24:24.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.214+0000 7f707e7fc640 1 -- 172.21.15.67:0/981281517 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70800b4310 con 0x7f7080149dc0 2024-03-29T11:24:24.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 1 -- 172.21.15.67:0/981281517 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4010440256 0 0) 0x7f70800afdd0 con 0x7f70800a9930 2024-03-29T11:24:24.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 1 -- 172.21.15.67:0/981281517 >> v1:172.21.15.67:6790/0 conn(0x7f708014ff40 legacy=0x7f7080152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 1 -- 172.21.15.67:0/981281517 >> v1:172.21.15.67:6789/0 conn(0x7f7080149dc0 legacy=0x7f708014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:24.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:24:24.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:24.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:24:24.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 1 -- 172.21.15.67:0/981281517 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708014a950 con 0x7f70800a9930 2024-03-29T11:24:24.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:24.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:24.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:24.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 1 -- 172.21.15.67:0/981281517 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7070001520 con 0x7f70800a9930 2024-03-29T11:24:24.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:24.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.215+0000 7f707e7fc640 10 monclient: dump: 2024-03-29T11:24:24.199 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:24.199 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:24.199 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:24.199 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:24.199 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.199 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:24.199 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:24.199 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:24.199 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:24.199 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:24.200 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.216+0000 7f708611c640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:24:24.200 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.216+0000 7f708611c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:24.200 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.217+0000 7f707e7fc640 1 -- 172.21.15.67:0/981281517 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7070001680 con 0x7f70800a9930 2024-03-29T11:24:24.200 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.217+0000 7f707e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:24.200 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.217+0000 7f707e7fc640 1 -- 172.21.15.67:0/981281517 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7070001ad0 con 0x7f70800a9930 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.217+0000 7f707e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.217+0000 7f707e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.217+0000 7f707e7fc640 10 monclient: dump: 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.217+0000 7f708611c640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:24.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.217+0000 7f708611c640 4 set_mon_vals no callback set 2024-03-29T11:24:24.202 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.218+0000 7f708611c640 10 monclient: shutdown 2024-03-29T11:24:24.202 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.218+0000 7f708611c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:24.202 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.218+0000 7f708611c640 1 -- 172.21.15.67:0/981281517 >> v1:172.21.15.82:6789/0 conn(0x7f70800a9930 legacy=0x7f7080143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.202 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.219+0000 7f708611c640 1 -- 172.21.15.67:0/981281517 shutdown_connections 2024-03-29T11:24:24.202 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.219+0000 7f708611c640 1 -- 172.21.15.67:0/981281517 >> 172.21.15.67:0/981281517 conn(0x7f70800b35e0 msgr2=0x7f70800b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:24.202 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.219+0000 7f708611c640 1 -- 172.21.15.67:0/981281517 shutdown_connections 2024-03-29T11:24:24.202 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.219+0000 7f708611c640 1 -- 172.21.15.67:0/981281517 wait complete. 2024-03-29T11:24:24.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.220+0000 7f708611c640 10 monclient: build_initial_monmap 2024-03-29T11:24:24.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.220+0000 7f708611c640 10 monclient: monmap: 2024-03-29T11:24:24.205 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:24.205 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:24.205 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:24:24.205 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:24:24.205 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:24.205 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:24.205 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:24.205 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:24:24.205 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:24:24.205 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:24.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.220+0000 7f708611c640 1 librados: starting msgr at 2024-03-29T11:24:24.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.220+0000 7f708611c640 1 librados: starting objecter 2024-03-29T11:24:24.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.220+0000 7f708611c640 1 Processor -- start 2024-03-29T11:24:24.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.220+0000 7f708611c640 1 -- start start 2024-03-29T11:24:24.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.220+0000 7f708611c640 1 librados: setting wanted keys 2024-03-29T11:24:24.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.220+0000 7f708611c640 1 librados: calling monclient init 2024-03-29T11:24:24.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.220+0000 7f708611c640 10 monclient: init 2024-03-29T11:24:24.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.221+0000 7f708611c640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:24.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.221+0000 7f708611c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:24.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.221+0000 7f708611c640 10 monclient(hunting): picked mon.noname-c con 0x7f70800a9930 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:24.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.221+0000 7f708611c640 10 monclient(hunting): picked mon.noname-a con 0x7f7080149dc0 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:24.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.221+0000 7f708611c640 10 monclient(hunting): picked mon.noname-b con 0x7f708014ff40 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:24.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.222+0000 7f708611c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70801dc660 con 0x7f7080149dc0 2024-03-29T11:24:24.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.222+0000 7f708611c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70801dba30 con 0x7f708014ff40 2024-03-29T11:24:24.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.222+0000 7f708611c640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708013e890 con 0x7f70800a9930 2024-03-29T11:24:24.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.222+0000 7f708611c640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:24.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.222+0000 7f708611c640 10 monclient(hunting): authenticate will time out at 1124.518434s 2024-03-29T11:24:24.207 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.222+0000 7f707f7fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f70800a9930 0x7f70801dbf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:48986/0 (socket says 172.21.15.67:48986) 2024-03-29T11:24:24.207 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.222+0000 7f707f7fe640 1 -- 172.21.15.67:0/131448885 learned_addr learned my addr 172.21.15.67:0/131448885 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:24.207 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.222+0000 7f7067fff640 10 client.?.objecter ms_handle_connect 0x7f70800a9930 2024-03-29T11:24:24.207 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.222+0000 7f7067fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.223+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040969125 0 0) 0x7f708013e890 con 0x7f70800a9930 2024-03-29T11:24:24.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.223+0000 7f7067fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.223+0000 7f7067fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.223+0000 7f7067fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.223+0000 7f7067fff640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:24:24.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.223+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f706c004eb0 con 0x7f70800a9930 2024-03-29T11:24:24.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.224+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4249250155 0 0) 0x7f706c004eb0 con 0x7f70800a9930 2024-03-29T11:24:24.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.224+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f708013e890 con 0x7f70800a9930 2024-03-29T11:24:24.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.224+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7070001a90 con 0x7f70800a9930 2024-03-29T11:24:24.211 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.224+0000 7f7067fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.212 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.224+0000 7f7067fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:24.212 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.224+0000 7f7067fff640 10 monclient(hunting): dump: 2024-03-29T11:24:24.212 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:24.212 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:24.212 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:24.212 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:24.212 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.212 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:24.212 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:24.212 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:24.212 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:24.212 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:24.212 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.224+0000 7f7067fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:24.212 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.224+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2589768977 0 0) 0x7f708013e890 con 0x7f70800a9930 2024-03-29T11:24:24.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.224+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 >> v1:172.21.15.82:6789/0 conn(0x7f708014ff40 legacy=0x7f708013a2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:24.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.225+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 >> v1:172.21.15.67:6789/0 conn(0x7f7080149dc0 legacy=0x7f708013f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.225+0000 7f7067fff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:24.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.225+0000 7f7067fff640 1 monclient: found mon.c 2024-03-29T11:24:24.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.225+0000 7f7067fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:24.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.225+0000 7f7067fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:24.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.225+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708013ea20 con 0x7f70800a9930 2024-03-29T11:24:24.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.225+0000 7f7067fff640 10 monclient: _finish_auth 0 2024-03-29T11:24:24.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.225+0000 7f7067fff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:24.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.225+0000 7f7067fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:24.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.226+0000 7f708611c640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:24:24.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.226+0000 7f708611c640 10 monclient: _renew_subs 2024-03-29T11:24:24.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.226+0000 7f708611c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:24.213 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.226+0000 7f708611c640 1 -- 172.21.15.67:0/131448885 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f708013fb90 con 0x7f70800a9930 2024-03-29T11:24:24.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.226+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70700038a0 con 0x7f70800a9930 2024-03-29T11:24:24.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.226+0000 7f7067fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:24.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.226+0000 7f707dffb640 4 set_mon_vals no callback set 2024-03-29T11:24:24.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.226+0000 7f708611c640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:24.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.226+0000 7f708611c640 10 monclient: _renew_subs 2024-03-29T11:24:24.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.226+0000 7f708611c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:24.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.226+0000 7f708611c640 1 -- 172.21.15.67:0/131448885 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f708013abb0 con 0x7f70800a9930 2024-03-29T11:24:24.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.226+0000 7f708611c640 1 librados: init done 2024-03-29T11:24:24.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.227+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7070001a90 con 0x7f70800a9930 2024-03-29T11:24:24.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.227+0000 7f7067fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.227+0000 7f7067fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:24.214 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.227+0000 7f7067fff640 10 monclient: dump: 2024-03-29T11:24:24.214 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:24.215 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:24.215 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:24.215 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:24.215 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.215 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:24.215 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:24.215 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:24.215 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:24.215 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:24.215 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.228+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f7070003b10 con 0x7f70800a9930 2024-03-29T11:24:24.217 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.229+0000 7f707effd640 10 monclient: get_auth_request con 0x7f706c06dd10 auth_method 0 2024-03-29T11:24:24.217 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.230+0000 7f708611c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.217 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.231+0000 7f708611c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.217 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.231+0000 7f708611c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:24.217 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.231+0000 7f708611c640 1 -- 172.21.15.67:0/131448885 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70801db870 con 0x7f70800a9930 2024-03-29T11:24:24.217 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.231+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f7070054810 con 0x7f70800a9930 2024-03-29T11:24:24.217 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.231+0000 7f7067fff640 10 client.4436.objecter ms_dispatch 0x7f7080000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:24.217 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.231+0000 7f7067fff640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:24.217 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.231+0000 7f7067fff640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:24.217 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.231+0000 7f7067fff640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-29T11:24:24.217 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.232+0000 7f7067fff640 10 client.4436.objecter ms_handle_connect 0x7f706c06dd10 2024-03-29T11:24:24.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.240+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7070050ab0 con 0x7f70800a9930 2024-03-29T11:24:24.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.240+0000 7f7067fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.240+0000 7f7067fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:24.549 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.567+0000 7f708611c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:24.549 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.567+0000 7f708611c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:24.549 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.567+0000 7f708611c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:24.549 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.567+0000 7f708611c640 1 -- 172.21.15.67:0/131448885 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f708013fd20 con 0x7f70800a9930 2024-03-29T11:24:24.549 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.568+0000 7f7067fff640 1 -- 172.21.15.67:0/131448885 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f7070050ab0 con 0x7f70800a9930 2024-03-29T11:24:24.550 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.568+0000 7f7067fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:24.550 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.568+0000 7f7067fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:24.550 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:24:24.550 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:24.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.572+0000 7f708611c640 10 librados: watch_flush enter 2024-03-29T11:24:24.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.572+0000 7f708611c640 10 librados: watch_flush exit 2024-03-29T11:24:24.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.572+0000 7f708611c640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-29T11:24:24.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.572+0000 7f708611c640 10 client.4436.objecter successfully canceled tick 2024-03-29T11:24:24.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.572+0000 7f708611c640 1 -- 172.21.15.67:0/131448885 >> v1:172.21.15.67:6816/404787197 conn(0x7f706c06dd10 legacy=0x7f706c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.572+0000 7f708611c640 10 monclient: shutdown 2024-03-29T11:24:24.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.572+0000 7f708611c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:24.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.572+0000 7f708611c640 1 -- 172.21.15.67:0/131448885 >> v1:172.21.15.67:6790/0 conn(0x7f70800a9930 legacy=0x7f70801dbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.572+0000 7f708611c640 1 -- 172.21.15.67:0/131448885 shutdown_connections 2024-03-29T11:24:24.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.572+0000 7f708611c640 1 -- 172.21.15.67:0/131448885 >> 172.21.15.67:0/131448885 conn(0x7f70800b35e0 msgr2=0x7f7080140520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:24.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.573+0000 7f708611c640 1 -- 172.21.15.67:0/131448885 shutdown_connections 2024-03-29T11:24:24.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.573+0000 7f708611c640 1 -- 172.21.15.67:0/131448885 wait complete. 2024-03-29T11:24:24.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:24.573+0000 7f708611c640 1 librados: shutdown 2024-03-29T11:24:24.571 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:31.378 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:31.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.543+0000 7fddbb73b640 10 monclient: get_monmap_and_config 2024-03-29T11:24:31.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.543+0000 7fddbb73b640 10 monclient: build_initial_monmap 2024-03-29T11:24:31.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.543+0000 7fddbb73b640 10 monclient: monmap: 2024-03-29T11:24:31.525 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:31.525 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:31.525 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:24:31.544282+0000 2024-03-29T11:24:31.525 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:24:31.544282+0000 2024-03-29T11:24:31.525 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:31.525 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:31.526 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:31.526 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:24:31.526 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:24:31.526 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:31.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.545+0000 7fddbb73b640 1 Processor -- start 2024-03-29T11:24:31.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.545+0000 7fddbb73b640 1 -- start start 2024-03-29T11:24:31.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.545+0000 7fddbb73b640 10 monclient: init 2024-03-29T11:24:31.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.546+0000 7fddbb73b640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:31.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.546+0000 7fddbb73b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:31.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.546+0000 7fddbb73b640 10 monclient(hunting): picked mon.noname-c con 0x7fddb40b4790 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:31.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.546+0000 7fddbb73b640 10 monclient(hunting): picked mon.noname-a con 0x7fddb40aea90 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:31.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.546+0000 7fddbb73b640 10 monclient(hunting): picked mon.noname-b con 0x7fddb40b39f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:31.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.546+0000 7fddbb73b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb40b0c60 con 0x7fddb40aea90 2024-03-29T11:24:31.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.546+0000 7fddbb73b640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb40af020 con 0x7fddb40b4790 2024-03-29T11:24:31.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.546+0000 7fddbb73b640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb413fbd0 con 0x7fddb40b39f0 2024-03-29T11:24:31.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.546+0000 7fddbb73b640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:31.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.546+0000 7fddbb73b640 10 monclient(hunting): authenticate will time out at 1131.842231s 2024-03-29T11:24:31.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.547+0000 7fddb9cd8640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fddb40b39f0 0x7fddb413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:34394/0 (socket says 172.21.15.67:34394) 2024-03-29T11:24:31.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.547+0000 7fddb9cd8640 1 -- 172.21.15.67:0/2985996611 learned_addr learned my addr 172.21.15.67:0/2985996611 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:31.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.547+0000 7fddabfff640 1 -- 172.21.15.67:0/2985996611 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362690366 0 0) 0x7fddb413fbd0 con 0x7fddb40b39f0 2024-03-29T11:24:31.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.547+0000 7fddabfff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:31.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.547+0000 7fddabfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:31.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.547+0000 7fddabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:31.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.547+0000 7fddabfff640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:24:31.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.547+0000 7fddabfff640 1 -- 172.21.15.67:0/2985996611 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda0003520 con 0x7fddb40b39f0 2024-03-29T11:24:31.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.548+0000 7fddabfff640 1 -- 172.21.15.67:0/2985996611 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4093610804 0 0) 0x7fdda0003520 con 0x7fddb40b39f0 2024-03-29T11:24:31.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.548+0000 7fddabfff640 1 -- 172.21.15.67:0/2985996611 >> v1:172.21.15.82:6789/0 conn(0x7fddb40b4790 legacy=0x7fddb40b1080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:31.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.549+0000 7fddabfff640 1 -- 172.21.15.67:0/2985996611 >> v1:172.21.15.67:6789/0 conn(0x7fddb40aea90 legacy=0x7fddb40af540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:31.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.550+0000 7fddabfff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.550+0000 7fddabfff640 1 monclient: found mon.noname-b 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.550+0000 7fddabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.550+0000 7fddabfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.550+0000 7fddabfff640 1 -- 172.21.15.67:0/2985996611 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb413fd60 con 0x7fddb40b39f0 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.550+0000 7fddabfff640 10 monclient: _finish_auth 0 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.550+0000 7fddabfff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.550+0000 7fddabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.550+0000 7fddabfff640 1 -- 172.21.15.67:0/2985996611 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fddac001a10 con 0x7fddb40b39f0 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.550+0000 7fddabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.550+0000 7fddabfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.550+0000 7fddabfff640 10 monclient: dump: 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:31.532 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:31.533 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:31.533 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:31.533 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:31.533 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:31.533 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:31.533 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:31.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.550+0000 7fddbb73b640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:24:31.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.550+0000 7fddbb73b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:31.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.551+0000 7fddabfff640 1 -- 172.21.15.67:0/2985996611 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddac001be0 con 0x7fddb40b39f0 2024-03-29T11:24:31.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.551+0000 7fddabfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:31.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.551+0000 7fddabfff640 1 -- 172.21.15.67:0/2985996611 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fddac002030 con 0x7fddb40b39f0 2024-03-29T11:24:31.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.551+0000 7fddabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:31.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.551+0000 7fddabfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:31.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.551+0000 7fddabfff640 10 monclient: dump: 2024-03-29T11:24:31.534 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:31.534 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:31.534 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:31.535 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:31.535 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:31.535 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:31.535 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:31.535 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:31.535 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:31.535 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:31.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.551+0000 7fddbb73b640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:31.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.551+0000 7fddbb73b640 4 set_mon_vals no callback set 2024-03-29T11:24:31.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.552+0000 7fddbb73b640 10 monclient: shutdown 2024-03-29T11:24:31.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.552+0000 7fddbb73b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:31.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.552+0000 7fddbb73b640 1 -- 172.21.15.67:0/2985996611 >> v1:172.21.15.67:6790/0 conn(0x7fddb40b39f0 legacy=0x7fddb413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:31.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.552+0000 7fddbb73b640 1 -- 172.21.15.67:0/2985996611 shutdown_connections 2024-03-29T11:24:31.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.552+0000 7fddbb73b640 1 -- 172.21.15.67:0/2985996611 >> 172.21.15.67:0/2985996611 conn(0x7fddb4066a00 msgr2=0x7fddb4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:31.536 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.553+0000 7fddbb73b640 1 -- 172.21.15.67:0/2985996611 shutdown_connections 2024-03-29T11:24:31.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.553+0000 7fddbb73b640 1 -- 172.21.15.67:0/2985996611 wait complete. 2024-03-29T11:24:31.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.554+0000 7fddbb73b640 10 monclient: build_initial_monmap 2024-03-29T11:24:31.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.554+0000 7fddbb73b640 10 monclient: monmap: 2024-03-29T11:24:31.537 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:31.537 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:31.537 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:24:31.537 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:24:31.537 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:31.537 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:31.537 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:31.537 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:24:31.537 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:24:31.537 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:31.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.555+0000 7fddbb73b640 1 librados: starting msgr at 2024-03-29T11:24:31.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.555+0000 7fddbb73b640 1 librados: starting objecter 2024-03-29T11:24:31.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.555+0000 7fddbb73b640 1 Processor -- start 2024-03-29T11:24:31.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.558+0000 7fddbb73b640 1 -- start start 2024-03-29T11:24:31.540 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.558+0000 7fddbb73b640 1 librados: setting wanted keys 2024-03-29T11:24:31.540 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.558+0000 7fddbb73b640 1 librados: calling monclient init 2024-03-29T11:24:31.540 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.558+0000 7fddbb73b640 10 monclient: init 2024-03-29T11:24:31.540 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.559+0000 7fddbb73b640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:31.540 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.559+0000 7fddbb73b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:31.540 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.559+0000 7fddbb73b640 10 monclient(hunting): picked mon.noname-c con 0x7fddb40b39f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:31.540 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.559+0000 7fddbb73b640 10 monclient(hunting): picked mon.noname-b con 0x7fddb40b4790 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:31.540 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.559+0000 7fddbb73b640 10 monclient(hunting): picked mon.noname-a con 0x7fddb412ba40 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:31.541 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.559+0000 7fddbb73b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb4142bf0 con 0x7fddb412ba40 2024-03-29T11:24:31.541 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.559+0000 7fddbb73b640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb4143080 con 0x7fddb40b4790 2024-03-29T11:24:31.541 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.559+0000 7fddbb73b640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb4127b00 con 0x7fddb40b39f0 2024-03-29T11:24:31.541 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.559+0000 7fddbb73b640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:31.541 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.559+0000 7fddbb73b640 10 monclient(hunting): authenticate will time out at 1131.855164s 2024-03-29T11:24:31.551 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.562+0000 7fddb8cd6640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fddb40b4790 0x7fddb4129260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:37323/0 (socket says 172.21.15.67:37323) 2024-03-29T11:24:31.551 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.562+0000 7fddb8cd6640 1 -- 172.21.15.67:0/2755414207 learned_addr learned my addr 172.21.15.67:0/2755414207 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:31.551 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 client.?.objecter ms_handle_connect 0x7fddb40b39f0 2024-03-29T11:24:31.551 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:31.551 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 client.?.objecter ms_handle_connect 0x7fddb412ba40 2024-03-29T11:24:31.551 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:31.551 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 client.?.objecter ms_handle_connect 0x7fddb40b4790 2024-03-29T11:24:31.551 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:31.551 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728787532 0 0) 0x7fddb4142bf0 con 0x7fddb412ba40 2024-03-29T11:24:31.551 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:31.551 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:31.551 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:31.551 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:24:31.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd98004eb0 con 0x7fddb412ba40 2024-03-29T11:24:31.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926144449 0 0) 0x7fddb4127b00 con 0x7fddb40b39f0 2024-03-29T11:24:31.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:31.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:31.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:31.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:24:31.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb4142bf0 con 0x7fddb40b39f0 2024-03-29T11:24:31.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789145361 0 0) 0x7fddb4143080 con 0x7fddb40b4790 2024-03-29T11:24:31.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:31.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:31.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:31.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:24:31.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb4127b00 con 0x7fddb40b4790 2024-03-29T11:24:31.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2793036064 0 0) 0x7fddb4142bf0 con 0x7fddb40b39f0 2024-03-29T11:24:31.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddb4143080 con 0x7fddb40b39f0 2024-03-29T11:24:31.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fddb0001770 con 0x7fddb40b39f0 2024-03-29T11:24:31.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:31.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:31.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): dump: 2024-03-29T11:24:31.553 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:31.554 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:31.554 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:31.554 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:31.554 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:31.554 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:31.554 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:31.554 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:31.554 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:31.554 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:31.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:31.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2434320745 0 0) 0x7fddb4127b00 con 0x7fddb40b4790 2024-03-29T11:24:31.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.563+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddb4142bf0 con 0x7fddb40b4790 2024-03-29T11:24:31.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fdda4003e70 con 0x7fddb40b4790 2024-03-29T11:24:31.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:31.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:31.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 10 monclient(hunting): dump: 2024-03-29T11:24:31.555 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:31.555 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:31.555 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:31.555 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:31.555 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:31.555 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:31.555 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 494222249 0 0) 0x7fdd98004eb0 con 0x7fddb412ba40 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddb4127b00 con 0x7fddb412ba40 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fddac001ef0 con 0x7fddb412ba40 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 10 monclient(hunting): dump: 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:31.556 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:31.557 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:31.557 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:31.557 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:31.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 762333824 0 0) 0x7fddb4143080 con 0x7fddb40b39f0 2024-03-29T11:24:31.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 >> v1:172.21.15.82:6789/0 conn(0x7fddb40b4790 legacy=0x7fddb4129260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:31.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 >> v1:172.21.15.67:6789/0 conn(0x7fddb412ba40 legacy=0x7fddb41e3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:31.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:31.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 1 monclient: found mon.c 2024-03-29T11:24:31.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:31.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:31.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb4141c70 con 0x7fddb40b39f0 2024-03-29T11:24:31.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:31.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:31.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:31.558 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddb0001c60 con 0x7fddb40b39f0 2024-03-29T11:24:31.558 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:31.558 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fddb0003790 con 0x7fddb40b39f0 2024-03-29T11:24:31.558 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:31.558 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:31.558 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fdda9ffb640 10 monclient: dump: 2024-03-29T11:24:31.558 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:31.558 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:31.558 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:31.558 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:31.558 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:31.558 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:31.558 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:31.558 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:31.559 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:31.559 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:31.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.564+0000 7fddaaffd640 4 set_mon_vals no callback set 2024-03-29T11:24:31.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.565+0000 7fddbb73b640 5 monclient: authenticate success, global_id 4442 2024-03-29T11:24:31.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.565+0000 7fddbb73b640 10 monclient: _renew_subs 2024-03-29T11:24:31.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.565+0000 7fddbb73b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:31.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.565+0000 7fddbb73b640 1 -- 172.21.15.67:0/2755414207 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb4135540 con 0x7fddb40b39f0 2024-03-29T11:24:31.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.565+0000 7fddbb73b640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:31.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.565+0000 7fddbb73b640 10 monclient: _renew_subs 2024-03-29T11:24:31.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.565+0000 7fddbb73b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:31.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.565+0000 7fddbb73b640 1 -- 172.21.15.67:0/2755414207 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb4140680 con 0x7fddb40b39f0 2024-03-29T11:24:31.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.565+0000 7fddbb73b640 1 librados: init done 2024-03-29T11:24:31.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.568+0000 7fddbb73b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:31.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.568+0000 7fddbb73b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:31.560 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.568+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fddb0019760 con 0x7fddb40b39f0 2024-03-29T11:24:31.560 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.568+0000 7fddbb73b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:31.560 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.568+0000 7fddbb73b640 1 -- 172.21.15.67:0/2755414207 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb4067fc0 con 0x7fddb40b39f0 2024-03-29T11:24:31.560 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.569+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fddb00836d0 con 0x7fddb40b39f0 2024-03-29T11:24:31.560 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.569+0000 7fdda9ffb640 10 client.4442.objecter ms_dispatch 0x7fddb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:31.560 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.569+0000 7fdda9ffb640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:31.561 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.569+0000 7fdda9ffb640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:31.561 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.569+0000 7fdda9ffb640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-29T11:24:31.561 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.569+0000 7fddb8cd6640 10 monclient: get_auth_request con 0x7fdd9806e0f0 auth_method 0 2024-03-29T11:24:31.561 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.569+0000 7fdda9ffb640 10 client.4442.objecter ms_handle_connect 0x7fdd9806e0f0 2024-03-29T11:24:31.561 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.577+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fddb4067fc0 con 0x7fddb40b39f0 2024-03-29T11:24:31.561 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.577+0000 7fdda9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:31.561 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.577+0000 7fdda9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:31.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.909+0000 7fddbb73b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:31.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.909+0000 7fddbb73b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:31.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.909+0000 7fddbb73b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:31.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.909+0000 7fddbb73b640 1 -- 172.21.15.67:0/2755414207 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddb4067fc0 con 0x7fddb40b39f0 2024-03-29T11:24:31.892 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:24:31.892 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:31.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.910+0000 7fdda9ffb640 1 -- 172.21.15.67:0/2755414207 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fddb4067fc0 con 0x7fddb40b39f0 2024-03-29T11:24:31.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.910+0000 7fdda9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:31.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.910+0000 7fdda9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:31.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.914+0000 7fddbb73b640 10 librados: watch_flush enter 2024-03-29T11:24:31.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.914+0000 7fddbb73b640 10 librados: watch_flush exit 2024-03-29T11:24:31.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.914+0000 7fddbb73b640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-29T11:24:31.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.914+0000 7fddbb73b640 10 client.4442.objecter successfully canceled tick 2024-03-29T11:24:31.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.914+0000 7fddbb73b640 1 -- 172.21.15.67:0/2755414207 >> v1:172.21.15.67:6816/404787197 conn(0x7fdd9806e0f0 legacy=0x7fdd980705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:31.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.914+0000 7fddbb73b640 10 monclient: shutdown 2024-03-29T11:24:31.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.915+0000 7fddbb73b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:31.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.915+0000 7fddbb73b640 1 -- 172.21.15.67:0/2755414207 >> v1:172.21.15.67:6790/0 conn(0x7fddb40b39f0 legacy=0x7fddb413ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:31.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.915+0000 7fddbb73b640 1 -- 172.21.15.67:0/2755414207 shutdown_connections 2024-03-29T11:24:31.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.915+0000 7fddbb73b640 1 -- 172.21.15.67:0/2755414207 >> 172.21.15.67:0/2755414207 conn(0x7fddb4066a00 msgr2=0x7fddb413f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:31.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.915+0000 7fddbb73b640 1 -- 172.21.15.67:0/2755414207 shutdown_connections 2024-03-29T11:24:31.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.918+0000 7fddbb73b640 1 -- 172.21.15.67:0/2755414207 wait complete. 2024-03-29T11:24:31.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:31.918+0000 7fddbb73b640 1 librados: shutdown 2024-03-29T11:24:31.916 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:38.723 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:38.876 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.894+0000 7fe72ed74640 10 monclient: get_monmap_and_config 2024-03-29T11:24:38.877 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.894+0000 7fe72ed74640 10 monclient: build_initial_monmap 2024-03-29T11:24:38.877 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.894+0000 7fe72ed74640 10 monclient: monmap: 2024-03-29T11:24:38.877 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:38.877 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:38.877 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:24:38.895451+0000 2024-03-29T11:24:38.877 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:24:38.895451+0000 2024-03-29T11:24:38.877 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:38.877 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:38.877 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:38.877 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:24:38.877 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:24:38.877 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:38.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.895+0000 7fe72ed74640 1 Processor -- start 2024-03-29T11:24:38.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.895+0000 7fe72ed74640 1 -- start start 2024-03-29T11:24:38.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.895+0000 7fe72ed74640 10 monclient: init 2024-03-29T11:24:38.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.896+0000 7fe72ed74640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:38.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.896+0000 7fe72ed74640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:38.879 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.896+0000 7fe72ed74640 10 monclient(hunting): picked mon.noname-a con 0x7fe720006a70 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:38.879 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.896+0000 7fe72ed74640 10 monclient(hunting): picked mon.noname-c con 0x7fe72002b100 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:38.879 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.896+0000 7fe72ed74640 10 monclient(hunting): picked mon.noname-b con 0x7fe720029c30 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:38.879 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.896+0000 7fe72ed74640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72002a270 con 0x7fe720006a70 2024-03-29T11:24:38.879 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.896+0000 7fe72ed74640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72002c440 con 0x7fe72002b100 2024-03-29T11:24:38.879 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.896+0000 7fe72ed74640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7200c9440 con 0x7fe720029c30 2024-03-29T11:24:38.879 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.896+0000 7fe72ed74640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:38.879 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.896+0000 7fe72ed74640 10 monclient(hunting): authenticate will time out at 1139.192856s 2024-03-29T11:24:38.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.900+0000 7fe72dd72640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe720006a70 0x7fe7200c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39203/0 (socket says 172.21.15.67:39203) 2024-03-29T11:24:38.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.900+0000 7fe72dd72640 1 -- 172.21.15.67:0/458133831 learned_addr learned my addr 172.21.15.67:0/458133831 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:38.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 1 -- 172.21.15.67:0/458133831 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652381480 0 0) 0x7fe72002a270 con 0x7fe720006a70 2024-03-29T11:24:38.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 10 monclient(hunting): _negotiate 2024-03-29T11:24:38.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:38.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:38.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:24:38.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 1 -- 172.21.15.67:0/458133831 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe714003520 con 0x7fe720006a70 2024-03-29T11:24:38.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 1 -- 172.21.15.67:0/458133831 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 148372011 0 0) 0x7fe714003520 con 0x7fe720006a70 2024-03-29T11:24:38.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 1 -- 172.21.15.67:0/458133831 >> v1:172.21.15.67:6790/0 conn(0x7fe720029c30 legacy=0x7fe7200d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:38.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 1 -- 172.21.15.67:0/458133831 >> v1:172.21.15.82:6789/0 conn(0x7fe72002b100 legacy=0x7fe7200cf100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:38.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 10 monclient: _finish_hunting 0 2024-03-29T11:24:38.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 1 monclient: found mon.noname-a 2024-03-29T11:24:38.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:38.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:24:38.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 1 -- 172.21.15.67:0/458133831 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7200c95d0 con 0x7fe720006a70 2024-03-29T11:24:38.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 10 monclient: _finish_auth 0 2024-03-29T11:24:38.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 10 monclient: _check_auth_tickets 2024-03-29T11:24:38.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:38.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 1 -- 172.21.15.67:0/458133831 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe72804f730 con 0x7fe720006a70 2024-03-29T11:24:38.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:38.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:38.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72cd70640 10 monclient: dump: 2024-03-29T11:24:38.889 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:38.889 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:38.890 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:38.890 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:38.890 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:38.890 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:38.890 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:38.890 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:38.890 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:38.890 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:38.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72ed74640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:24:38.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.901+0000 7fe72ed74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:38.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.902+0000 7fe72cd70640 1 -- 172.21.15.67:0/458133831 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7280008d0 con 0x7fe720006a70 2024-03-29T11:24:38.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.902+0000 7fe72cd70640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:38.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.902+0000 7fe72cd70640 1 -- 172.21.15.67:0/458133831 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe7280758b0 con 0x7fe720006a70 2024-03-29T11:24:38.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.902+0000 7fe72cd70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:38.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.902+0000 7fe72cd70640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:38.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.902+0000 7fe72cd70640 10 monclient: dump: 2024-03-29T11:24:38.891 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:38.891 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:38.891 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:38.891 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:38.891 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:38.891 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:38.891 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:38.891 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.902+0000 7fe72ed74640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.902+0000 7fe72ed74640 4 set_mon_vals no callback set 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.903+0000 7fe72ed74640 10 monclient: shutdown 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.903+0000 7fe72ed74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.903+0000 7fe72ed74640 1 -- 172.21.15.67:0/458133831 >> v1:172.21.15.67:6789/0 conn(0x7fe720006a70 legacy=0x7fe7200c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.904+0000 7fe72ed74640 1 -- 172.21.15.67:0/458133831 shutdown_connections 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.904+0000 7fe72ed74640 1 -- 172.21.15.67:0/458133831 >> 172.21.15.67:0/458133831 conn(0x7fe7200058f0 msgr2=0x7fe720005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.904+0000 7fe72ed74640 1 -- 172.21.15.67:0/458133831 shutdown_connections 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.904+0000 7fe72ed74640 1 -- 172.21.15.67:0/458133831 wait complete. 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.905+0000 7fe72ed74640 10 monclient: build_initial_monmap 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.905+0000 7fe72ed74640 10 monclient: monmap: 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:38.892 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:38.893 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:24:38.893 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:24:38.893 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:38.893 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:38.893 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:38.893 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:24:38.893 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:24:38.893 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:38.893 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.906+0000 7fe72ed74640 1 librados: starting msgr at 2024-03-29T11:24:38.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.906+0000 7fe72ed74640 1 librados: starting objecter 2024-03-29T11:24:38.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.906+0000 7fe72ed74640 1 Processor -- start 2024-03-29T11:24:38.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.906+0000 7fe72ed74640 1 -- start start 2024-03-29T11:24:38.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.906+0000 7fe72ed74640 1 librados: setting wanted keys 2024-03-29T11:24:38.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.906+0000 7fe72ed74640 1 librados: calling monclient init 2024-03-29T11:24:38.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.906+0000 7fe72ed74640 10 monclient: init 2024-03-29T11:24:38.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.907+0000 7fe72ed74640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:38.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.907+0000 7fe72ed74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:38.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.907+0000 7fe72ed74640 10 monclient(hunting): picked mon.noname-b con 0x7fe720006a70 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:38.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.908+0000 7fe72ed74640 10 monclient(hunting): picked mon.noname-c con 0x7fe720029c30 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:38.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.908+0000 7fe72ed74640 10 monclient(hunting): picked mon.noname-a con 0x7fe7200cb7c0 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:38.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.908+0000 7fe72ed74640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7200ccb30 con 0x7fe7200cb7c0 2024-03-29T11:24:38.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.908+0000 7fe72ed74640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7200cb2a0 con 0x7fe720006a70 2024-03-29T11:24:38.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.908+0000 7fe72ed74640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7200caf00 con 0x7fe720029c30 2024-03-29T11:24:38.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.908+0000 7fe72ed74640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:38.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.908+0000 7fe72ed74640 10 monclient(hunting): authenticate will time out at 1139.203972s 2024-03-29T11:24:38.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.913+0000 7fe72e573640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe7200cb7c0 0x7fe72016b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39487/0 (socket says 172.21.15.67:39487) 2024-03-29T11:24:38.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.913+0000 7fe72e573640 1 -- 172.21.15.67:0/2583794968 learned_addr learned my addr 172.21.15.67:0/2583794968 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:38.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 10 client.?.objecter ms_handle_connect 0x7fe720029c30 2024-03-29T11:24:38.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:38.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323521667 0 0) 0x7fe7200caf00 con 0x7fe720029c30 2024-03-29T11:24:38.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:38.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:38.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:38.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:24:38.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe70c004eb0 con 0x7fe720029c30 2024-03-29T11:24:38.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 10 client.?.objecter ms_handle_connect 0x7fe7200cb7c0 2024-03-29T11:24:38.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:38.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1550997608 0 0) 0x7fe70c004eb0 con 0x7fe720029c30 2024-03-29T11:24:38.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7200caf00 con 0x7fe720029c30 2024-03-29T11:24:38.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe718001870 con 0x7fe720029c30 2024-03-29T11:24:38.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:38.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:38.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 10 monclient(hunting): dump: 2024-03-29T11:24:38.900 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:38.900 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:38.900 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:38.900 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:38.900 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:38.900 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:38.900 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:38.900 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:38.901 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:38.901 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:38.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:38.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1358420676 0 0) 0x7fe7200caf00 con 0x7fe720029c30 2024-03-29T11:24:38.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 >> v1:172.21.15.82:6789/0 conn(0x7fe720006a70 legacy=0x7fe7200ca740 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:24:38.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 >> v1:172.21.15.67:6789/0 conn(0x7fe7200cb7c0 legacy=0x7fe72016b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:38.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:38.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 1 monclient: found mon.c 2024-03-29T11:24:38.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:38.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:38.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.915+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7200cdf80 con 0x7fe720029c30 2024-03-29T11:24:38.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe71e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:38.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe71e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:38.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe71e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:38.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe72ed74640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:24:38.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe72ed74640 10 monclient: _renew_subs 2024-03-29T11:24:38.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe72ed74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:38.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe72ed74640 1 -- 172.21.15.67:0/2583794968 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7200c9980 con 0x7fe720029c30 2024-03-29T11:24:38.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe72ed74640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:38.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe72ed74640 10 monclient: _renew_subs 2024-03-29T11:24:38.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe72ed74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:38.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe72ed74640 1 -- 172.21.15.67:0/2583794968 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72016ce40 con 0x7fe720029c30 2024-03-29T11:24:38.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe72ed74640 1 librados: init done 2024-03-29T11:24:38.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe718001b00 con 0x7fe720029c30 2024-03-29T11:24:38.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe71e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:38.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe7180038b0 con 0x7fe720029c30 2024-03-29T11:24:38.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe71e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:38.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe71e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:38.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe71e7fc640 10 monclient: dump: 2024-03-29T11:24:38.903 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:38.903 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:38.903 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:38.903 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:38.903 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:38.903 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:38.903 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:38.903 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:38.903 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:38.903 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:38.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.916+0000 7fe71ffff640 4 set_mon_vals no callback set 2024-03-29T11:24:38.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.919+0000 7fe72ed74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:38.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.920+0000 7fe72ed74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:38.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.920+0000 7fe72ed74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:38.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.920+0000 7fe72ed74640 1 -- 172.21.15.67:0/2583794968 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe710005080 con 0x7fe720029c30 2024-03-29T11:24:38.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.923+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fe718003a10 con 0x7fe720029c30 2024-03-29T11:24:38.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.923+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fe718083810 con 0x7fe720029c30 2024-03-29T11:24:38.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.923+0000 7fe71e7fc640 10 client.4445.objecter ms_dispatch 0x7fe728000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:38.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.923+0000 7fe71e7fc640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:38.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.923+0000 7fe71e7fc640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:38.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.923+0000 7fe71e7fc640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T11:24:38.909 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.927+0000 7fe72dd72640 10 monclient: get_auth_request con 0x7fe70c06dbd0 auth_method 0 2024-03-29T11:24:38.910 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.928+0000 7fe71e7fc640 10 client.4445.objecter ms_handle_connect 0x7fe70c06dbd0 2024-03-29T11:24:38.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.931+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe71804ef20 con 0x7fe720029c30 2024-03-29T11:24:38.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.931+0000 7fe71e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:38.913 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:38.931+0000 7fe71e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:39.265 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.283+0000 7fe72ed74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:39.266 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.283+0000 7fe72ed74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:39.266 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.283+0000 7fe72ed74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:39.266 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.283+0000 7fe72ed74640 1 -- 172.21.15.67:0/2583794968 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe710005330 con 0x7fe720029c30 2024-03-29T11:24:39.267 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.285+0000 7fe71e7fc640 1 -- 172.21.15.67:0/2583794968 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fe718051020 con 0x7fe720029c30 2024-03-29T11:24:39.267 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.285+0000 7fe71e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:39.267 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.285+0000 7fe71e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:39.269 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:24:39.269 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:39.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.288+0000 7fe72ed74640 10 librados: watch_flush enter 2024-03-29T11:24:39.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.289+0000 7fe72ed74640 10 librados: watch_flush exit 2024-03-29T11:24:39.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.289+0000 7fe72ed74640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T11:24:39.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.289+0000 7fe72ed74640 10 client.4445.objecter successfully canceled tick 2024-03-29T11:24:39.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.289+0000 7fe72ed74640 1 -- 172.21.15.67:0/2583794968 >> v1:172.21.15.67:6816/404787197 conn(0x7fe70c06dbd0 legacy=0x7fe70c070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.289+0000 7fe72ed74640 10 monclient: shutdown 2024-03-29T11:24:39.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.289+0000 7fe72ed74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:39.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.289+0000 7fe72ed74640 1 -- 172.21.15.67:0/2583794968 >> v1:172.21.15.67:6790/0 conn(0x7fe720029c30 legacy=0x7fe7200cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.289+0000 7fe72ed74640 1 -- 172.21.15.67:0/2583794968 shutdown_connections 2024-03-29T11:24:39.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.289+0000 7fe72ed74640 1 -- 172.21.15.67:0/2583794968 >> 172.21.15.67:0/2583794968 conn(0x7fe7200058f0 msgr2=0x7fe7200c3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:39.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.290+0000 7fe72ed74640 1 -- 172.21.15.67:0/2583794968 shutdown_connections 2024-03-29T11:24:39.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.291+0000 7fe72ed74640 1 -- 172.21.15.67:0/2583794968 wait complete. 2024-03-29T11:24:39.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:39.291+0000 7fe72ed74640 1 librados: shutdown 2024-03-29T11:24:39.291 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:46.098 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:46.266 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.283+0000 7f9642332640 10 monclient: get_monmap_and_config 2024-03-29T11:24:46.266 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.283+0000 7f9642332640 10 monclient: build_initial_monmap 2024-03-29T11:24:46.266 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.283+0000 7f9642332640 10 monclient: monmap: 2024-03-29T11:24:46.266 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:46.267 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:46.267 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:24:46.284901+0000 2024-03-29T11:24:46.267 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:24:46.284901+0000 2024-03-29T11:24:46.267 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:46.267 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:46.267 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:46.267 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:24:46.267 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:24:46.267 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:46.267 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.284+0000 7f9642332640 1 Processor -- start 2024-03-29T11:24:46.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.285+0000 7f9642332640 1 -- start start 2024-03-29T11:24:46.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.285+0000 7f9642332640 10 monclient: init 2024-03-29T11:24:46.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.285+0000 7f9642332640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:46.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.285+0000 7f9642332640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:46.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.286+0000 7f9642332640 10 monclient(hunting): picked mon.noname-c con 0x7f963c0b4330 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:46.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.286+0000 7f9642332640 10 monclient(hunting): picked mon.noname-b con 0x7f963c1470b0 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:46.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.286+0000 7f9642332640 10 monclient(hunting): picked mon.noname-a con 0x7f963c1439b0 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:46.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.286+0000 7f9642332640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c06f2a0 con 0x7f963c1439b0 2024-03-29T11:24:46.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.286+0000 7f9642332640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c146b90 con 0x7f963c0b4330 2024-03-29T11:24:46.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.286+0000 7f9642332640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c0b2ca0 con 0x7f963c1470b0 2024-03-29T11:24:46.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.286+0000 7f9642332640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:46.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.286+0000 7f9642332640 10 monclient(hunting): authenticate will time out at 1146.583016s 2024-03-29T11:24:46.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.288+0000 7f96337fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f963c1470b0 0x7f963c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:39686/0 (socket says 172.21.15.67:39686) 2024-03-29T11:24:46.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.288+0000 7f96337fe640 1 -- 172.21.15.67:0/3108292983 learned_addr learned my addr 172.21.15.67:0/3108292983 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:46.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.291+0000 7f963b7fe640 1 -- 172.21.15.67:0/3108292983 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742868688 0 0) 0x7f963c0b2ca0 con 0x7f963c1470b0 2024-03-29T11:24:46.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.291+0000 7f963b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:46.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.291+0000 7f963b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:46.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.291+0000 7f963b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:46.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.291+0000 7f963b7fe640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:24:46.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.291+0000 7f963b7fe640 1 -- 172.21.15.67:0/3108292983 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9628003520 con 0x7f963c1470b0 2024-03-29T11:24:46.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.291+0000 7f963b7fe640 1 -- 172.21.15.67:0/3108292983 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327462948 0 0) 0x7f963c06f2a0 con 0x7f963c1439b0 2024-03-29T11:24:46.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.291+0000 7f963b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:46.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.291+0000 7f963b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:46.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.291+0000 7f963b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:46.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.291+0000 7f963b7fe640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:24:46.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.291+0000 7f963b7fe640 1 -- 172.21.15.67:0/3108292983 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c0b2ca0 con 0x7f963c1439b0 2024-03-29T11:24:46.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 1 -- 172.21.15.67:0/3108292983 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3799737672 0 0) 0x7f963c0b2ca0 con 0x7f963c1439b0 2024-03-29T11:24:46.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 1 -- 172.21.15.67:0/3108292983 >> v1:172.21.15.67:6790/0 conn(0x7f963c1470b0 legacy=0x7f963c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:46.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 1 -- 172.21.15.67:0/3108292983 >> v1:172.21.15.82:6789/0 conn(0x7f963c0b4330 legacy=0x7f963c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:46.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:46.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:24:46.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:46.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:24:46.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 1 -- 172.21.15.67:0/3108292983 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c06e0c0 con 0x7f963c1439b0 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 1 -- 172.21.15.67:0/3108292983 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f96340018a0 con 0x7f963c1439b0 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 10 monclient: dump: 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:46.277 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:46.278 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:46.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 1 -- 172.21.15.67:0/3108292983 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9634001a00 con 0x7f963c1439b0 2024-03-29T11:24:46.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:46.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 1 -- 172.21.15.67:0/3108292983 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f9634001e10 con 0x7f963c1439b0 2024-03-29T11:24:46.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:46.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:46.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.292+0000 7f963b7fe640 10 monclient: dump: 2024-03-29T11:24:46.278 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:46.278 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:46.278 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:46.278 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:46.278 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:46.278 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:46.279 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:46.279 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:46.279 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:46.279 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:46.280 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.294+0000 7f9642332640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:24:46.280 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.294+0000 7f9642332640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:46.280 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.294+0000 7f9642332640 4 set_mon_vals no callback set 2024-03-29T11:24:46.280 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.295+0000 7f9642332640 10 monclient: shutdown 2024-03-29T11:24:46.280 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.295+0000 7f9642332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:46.280 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.295+0000 7f9642332640 1 -- 172.21.15.67:0/3108292983 >> v1:172.21.15.67:6789/0 conn(0x7f963c1439b0 legacy=0x7f963c151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:46.280 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.296+0000 7f9642332640 1 -- 172.21.15.67:0/3108292983 shutdown_connections 2024-03-29T11:24:46.281 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.296+0000 7f9642332640 1 -- 172.21.15.67:0/3108292983 >> 172.21.15.67:0/3108292983 conn(0x7f963c066a00 msgr2=0x7f963c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:46.281 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.297+0000 7f9642332640 1 -- 172.21.15.67:0/3108292983 shutdown_connections 2024-03-29T11:24:46.281 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.297+0000 7f9642332640 1 -- 172.21.15.67:0/3108292983 wait complete. 2024-03-29T11:24:46.284 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.298+0000 7f9642332640 10 monclient: build_initial_monmap 2024-03-29T11:24:46.285 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.298+0000 7f9642332640 10 monclient: monmap: 2024-03-29T11:24:46.285 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:46.285 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:46.285 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:24:46.285 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:24:46.285 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:46.285 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:46.285 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:46.285 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:24:46.285 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:24:46.285 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:46.285 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.298+0000 7f9642332640 1 librados: starting msgr at 2024-03-29T11:24:46.285 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.298+0000 7f9642332640 1 librados: starting objecter 2024-03-29T11:24:46.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.298+0000 7f9642332640 1 Processor -- start 2024-03-29T11:24:46.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.298+0000 7f9642332640 1 -- start start 2024-03-29T11:24:46.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.298+0000 7f9642332640 1 librados: setting wanted keys 2024-03-29T11:24:46.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.298+0000 7f9642332640 1 librados: calling monclient init 2024-03-29T11:24:46.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.298+0000 7f9642332640 10 monclient: init 2024-03-29T11:24:46.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.299+0000 7f9642332640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:46.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.299+0000 7f9642332640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:46.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.299+0000 7f9642332640 10 monclient(hunting): picked mon.noname-c con 0x7f963c0b4330 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:46.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.299+0000 7f9642332640 10 monclient(hunting): picked mon.noname-a con 0x7f963c1439b0 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:46.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.299+0000 7f9642332640 10 monclient(hunting): picked mon.noname-b con 0x7f963c1470b0 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:46.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.299+0000 7f9642332640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c121aa0 con 0x7f963c1439b0 2024-03-29T11:24:46.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.299+0000 7f9642332640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c1dbc10 con 0x7f963c1470b0 2024-03-29T11:24:46.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.299+0000 7f9642332640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c1df150 con 0x7f963c0b4330 2024-03-29T11:24:46.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.299+0000 7f9642332640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:46.288 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.299+0000 7f9642332640 10 monclient(hunting): authenticate will time out at 1146.596646s 2024-03-29T11:24:46.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.302+0000 7f963bfff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f963c0b4330 0x7f963c1dea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:42264/0 (socket says 172.21.15.67:42264) 2024-03-29T11:24:46.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.302+0000 7f963bfff640 1 -- 172.21.15.67:0/1191866728 learned_addr learned my addr 172.21.15.67:0/1191866728 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:46.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.302+0000 7f96397fa640 10 client.?.objecter ms_handle_connect 0x7f963c0b4330 2024-03-29T11:24:46.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.302+0000 7f96397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:46.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165916114 0 0) 0x7f963c1df150 con 0x7f963c0b4330 2024-03-29T11:24:46.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:46.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:46.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:46.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:24:46.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f961c004eb0 con 0x7f963c0b4330 2024-03-29T11:24:46.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1238055672 0 0) 0x7f961c004eb0 con 0x7f963c0b4330 2024-03-29T11:24:46.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f963c1df150 con 0x7f963c0b4330 2024-03-29T11:24:46.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f962c0017f0 con 0x7f963c0b4330 2024-03-29T11:24:46.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:46.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:46.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 10 monclient(hunting): dump: 2024-03-29T11:24:46.290 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:46.290 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:46.290 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:46.290 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 10 client.?.objecter ms_handle_connect 0x7f963c1470b0 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.303+0000 7f96397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.304+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281271889 0 0) 0x7f963c1dbc10 con 0x7f963c1470b0 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.304+0000 7f96397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.304+0000 7f96397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.304+0000 7f96397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.304+0000 7f96397fa640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:24:46.291 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.304+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f961c004eb0 con 0x7f963c1470b0 2024-03-29T11:24:46.292 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.304+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3300782705 0 0) 0x7f961c004eb0 con 0x7f963c1470b0 2024-03-29T11:24:46.292 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.305+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f963c1dbc10 con 0x7f963c1470b0 2024-03-29T11:24:46.292 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.305+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f9634001b70 con 0x7f963c1470b0 2024-03-29T11:24:46.292 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.305+0000 7f96397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:46.292 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.305+0000 7f96397fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:46.292 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.305+0000 7f96397fa640 10 monclient(hunting): dump: 2024-03-29T11:24:46.292 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:46.292 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:46.292 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:46.292 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:46.292 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:46.292 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:46.292 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:46.292 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:46.293 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:46.293 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:46.293 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.305+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 472043641 0 0) 0x7f963c1df150 con 0x7f963c0b4330 2024-03-29T11:24:46.293 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.305+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 >> v1:172.21.15.82:6789/0 conn(0x7f963c1470b0 legacy=0x7f963c1e6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:46.293 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.305+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 >> v1:172.21.15.67:6789/0 conn(0x7f963c1439b0 legacy=0x7f963c1405f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:46.293 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f96397fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:46.293 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f96397fa640 1 monclient: found mon.c 2024-03-29T11:24:46.294 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f96397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:46.294 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f96397fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:46.294 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c0b77e0 con 0x7f963c0b4330 2024-03-29T11:24:46.294 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f96397fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:46.294 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f96397fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:46.294 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f96397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:46.294 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f9642332640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:24:46.294 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f9642332640 10 monclient: _renew_subs 2024-03-29T11:24:46.294 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f9642332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:46.294 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f9642332640 1 -- 172.21.15.67:0/1191866728 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963c1dc130 con 0x7f963c0b4330 2024-03-29T11:24:46.294 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f9642332640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:46.294 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f9642332640 10 monclient: _renew_subs 2024-03-29T11:24:46.294 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f9642332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f9642332640 1 -- 172.21.15.67:0/1191866728 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f963c140d00 con 0x7f963c0b4330 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.306+0000 7f9642332640 1 librados: init done 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.307+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f962c001950 con 0x7f963c0b4330 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.307+0000 7f96397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.307+0000 7f963affd640 4 set_mon_vals no callback set 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.307+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f962c004a40 con 0x7f963c0b4330 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.307+0000 7f96397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.307+0000 7f96397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.307+0000 7f96397fa640 10 monclient: dump: 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:46.295 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:46.296 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:46.296 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:46.296 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:46.296 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:46.296 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.309+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f962c01aa10 con 0x7f963c0b4330 2024-03-29T11:24:46.297 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.310+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f962c084a50 con 0x7f963c0b4330 2024-03-29T11:24:46.297 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.310+0000 7f96397fa640 10 client.4451.objecter ms_dispatch 0x7f963c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:46.297 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.310+0000 7f96397fa640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:46.297 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.310+0000 7f96397fa640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:46.297 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.310+0000 7f9642332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:46.297 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.310+0000 7f9642332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:46.297 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.310+0000 7f9642332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:46.297 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.310+0000 7f96397fa640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T11:24:46.298 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.310+0000 7f9642332640 1 -- 172.21.15.67:0/1191866728 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9620005080 con 0x7f963c0b4330 2024-03-29T11:24:46.298 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.311+0000 7f96337fe640 10 monclient: get_auth_request con 0x7f961c06dee0 auth_method 0 2024-03-29T11:24:46.298 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.312+0000 7f96397fa640 10 client.4451.objecter ms_handle_connect 0x7f961c06dee0 2024-03-29T11:24:46.307 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.324+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f962c050160 con 0x7f963c0b4330 2024-03-29T11:24:46.307 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.324+0000 7f96397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:46.307 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.324+0000 7f96397fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:46.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.697+0000 7f9642332640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:46.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.697+0000 7f9642332640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:46.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.697+0000 7f9642332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:24:46.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.697+0000 7f9642332640 1 -- 172.21.15.67:0/1191866728 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9620005330 con 0x7f963c0b4330 2024-03-29T11:24:46.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.699+0000 7f96397fa640 1 -- 172.21.15.67:0/1191866728 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f962c052260 con 0x7f963c0b4330 2024-03-29T11:24:46.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.699+0000 7f96397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:46.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.699+0000 7f96397fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:46.684 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:24:46.684 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:46.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.703+0000 7f9642332640 10 librados: watch_flush enter 2024-03-29T11:24:46.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.703+0000 7f9642332640 10 librados: watch_flush exit 2024-03-29T11:24:46.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.703+0000 7f9642332640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T11:24:46.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.703+0000 7f9642332640 10 client.4451.objecter successfully canceled tick 2024-03-29T11:24:46.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.703+0000 7f9642332640 1 -- 172.21.15.67:0/1191866728 >> v1:172.21.15.67:6816/404787197 conn(0x7f961c06dee0 legacy=0x7f961c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:46.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.703+0000 7f9642332640 10 monclient: shutdown 2024-03-29T11:24:46.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.703+0000 7f9642332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:46.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.703+0000 7f9642332640 1 -- 172.21.15.67:0/1191866728 >> v1:172.21.15.67:6790/0 conn(0x7f963c0b4330 legacy=0x7f963c1dea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:46.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.703+0000 7f9642332640 1 -- 172.21.15.67:0/1191866728 shutdown_connections 2024-03-29T11:24:46.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.703+0000 7f9642332640 1 -- 172.21.15.67:0/1191866728 >> 172.21.15.67:0/1191866728 conn(0x7f963c066a00 msgr2=0x7f963c1d1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:46.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.704+0000 7f9642332640 1 -- 172.21.15.67:0/1191866728 shutdown_connections 2024-03-29T11:24:46.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.704+0000 7f9642332640 1 -- 172.21.15.67:0/1191866728 wait complete. 2024-03-29T11:24:46.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:46.704+0000 7f9642332640 1 librados: shutdown 2024-03-29T11:24:46.705 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:53.511 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:53.663 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.680+0000 7fb484bcd640 10 monclient: get_monmap_and_config 2024-03-29T11:24:53.663 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.680+0000 7fb484bcd640 10 monclient: build_initial_monmap 2024-03-29T11:24:53.663 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.680+0000 7fb484bcd640 10 monclient: monmap: 2024-03-29T11:24:53.663 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:53.664 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:53.664 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:24:53.681565+0000 2024-03-29T11:24:53.664 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:24:53.681565+0000 2024-03-29T11:24:53.664 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:53.664 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:53.664 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:53.664 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:24:53.664 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:24:53.664 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:53.665 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.681+0000 7fb484bcd640 1 Processor -- start 2024-03-29T11:24:53.665 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.681+0000 7fb484bcd640 1 -- start start 2024-03-29T11:24:53.665 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.681+0000 7fb484bcd640 10 monclient: init 2024-03-29T11:24:53.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb484bcd640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:53.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb484bcd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:53.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb484bcd640 10 monclient(hunting): picked mon.noname-a con 0x7fb4801457f0 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:53.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb484bcd640 10 monclient(hunting): picked mon.noname-b con 0x7fb48013f650 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:53.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb484bcd640 10 monclient(hunting): picked mon.noname-c con 0x7fb4800a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:53.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb484bcd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4800aaef0 con 0x7fb4801457f0 2024-03-29T11:24:53.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb484bcd640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb480145bd0 con 0x7fb4800a9d70 2024-03-29T11:24:53.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb484bcd640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb480145eb0 con 0x7fb48013f650 2024-03-29T11:24:53.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb484bcd640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:53.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb484bcd640 10 monclient(hunting): authenticate will time out at 1153.979085s 2024-03-29T11:24:53.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb47e59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb4801457f0 0x7fb48013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56930/0 (socket says 172.21.15.67:56930) 2024-03-29T11:24:53.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb47e59c640 1 -- 172.21.15.67:0/1469410518 learned_addr learned my addr 172.21.15.67:0/1469410518 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:53.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb47d59a640 1 -- 172.21.15.67:0/1469410518 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736752094 0 0) 0x7fb480145bd0 con 0x7fb4800a9d70 2024-03-29T11:24:53.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb47d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:53.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb47d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:53.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb47d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:53.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb47d59a640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:24:53.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.683+0000 7fb47d59a640 1 -- 172.21.15.67:0/1469410518 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb468003520 con 0x7fb4800a9d70 2024-03-29T11:24:53.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 1 -- 172.21.15.67:0/1469410518 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 927439353 0 0) 0x7fb468003520 con 0x7fb4800a9d70 2024-03-29T11:24:53.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 1 -- 172.21.15.67:0/1469410518 >> v1:172.21.15.67:6790/0 conn(0x7fb48013f650 legacy=0x7fb4800b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:53.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 1 -- 172.21.15.67:0/1469410518 >> v1:172.21.15.67:6789/0 conn(0x7fb4801457f0 legacy=0x7fb48013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:53.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 1 monclient: found mon.noname-c 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 1 -- 172.21.15.67:0/1469410518 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb480146340 con 0x7fb4800a9d70 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 10 monclient: _finish_auth 0 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 1 -- 172.21.15.67:0/1469410518 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb478001a10 con 0x7fb4800a9d70 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 10 monclient: dump: 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:53.671 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb484bcd640 5 monclient: authenticate success, global_id 4384 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb484bcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 1 -- 172.21.15.67:0/1469410518 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb478001b70 con 0x7fb4800a9d70 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 1 -- 172.21.15.67:0/1469410518 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb478001fc0 con 0x7fb4800a9d70 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb47d59a640 10 monclient: dump: 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:53.672 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb484bcd640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.684+0000 7fb484bcd640 4 set_mon_vals no callback set 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.685+0000 7fb484bcd640 10 monclient: shutdown 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.685+0000 7fb484bcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.685+0000 7fb484bcd640 1 -- 172.21.15.67:0/1469410518 >> v1:172.21.15.82:6789/0 conn(0x7fb4800a9d70 legacy=0x7fb48014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.685+0000 7fb484bcd640 1 -- 172.21.15.67:0/1469410518 shutdown_connections 2024-03-29T11:24:53.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.685+0000 7fb484bcd640 1 -- 172.21.15.67:0/1469410518 >> 172.21.15.67:0/1469410518 conn(0x7fb480066a00 msgr2=0x7fb480066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:53.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.686+0000 7fb484bcd640 1 -- 172.21.15.67:0/1469410518 shutdown_connections 2024-03-29T11:24:53.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.686+0000 7fb484bcd640 1 -- 172.21.15.67:0/1469410518 wait complete. 2024-03-29T11:24:53.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.687+0000 7fb484bcd640 10 monclient: build_initial_monmap 2024-03-29T11:24:53.678 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.687+0000 7fb484bcd640 10 monclient: monmap: 2024-03-29T11:24:53.678 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:24:53.678 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:53.678 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:24:53.678 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:24:53.678 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.687+0000 7fb484bcd640 1 librados: starting msgr at 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.687+0000 7fb484bcd640 1 librados: starting objecter 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.687+0000 7fb484bcd640 1 Processor -- start 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.687+0000 7fb484bcd640 1 -- start start 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.687+0000 7fb484bcd640 1 librados: setting wanted keys 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.687+0000 7fb484bcd640 1 librados: calling monclient init 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.687+0000 7fb484bcd640 10 monclient: init 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.688+0000 7fb484bcd640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.688+0000 7fb484bcd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.688+0000 7fb484bcd640 10 monclient(hunting): picked mon.noname-a con 0x7fb4800a9d70 addr v1:172.21.15.67:6789/0 2024-03-29T11:24:53.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.688+0000 7fb484bcd640 10 monclient(hunting): picked mon.noname-c con 0x7fb48013f650 addr v1:172.21.15.67:6790/0 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb484bcd640 10 monclient(hunting): picked mon.noname-b con 0x7fb4801457f0 addr v1:172.21.15.82:6789/0 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb484bcd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb480070620 con 0x7fb4800a9d70 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb484bcd640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48013a690 con 0x7fb4801457f0 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb484bcd640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb480071ce0 con 0x7fb48013f650 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb484bcd640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb484bcd640 10 monclient(hunting): authenticate will time out at 1153.985054s 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb47ed9d640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fb4801457f0 0x7fb480139f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:41210/0 (socket says 172.21.15.67:41210) 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb47ed9d640 1 -- 172.21.15.67:0/1242323859 learned_addr learned my addr 172.21.15.67:0/1242323859 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb46effd640 10 client.?.objecter ms_handle_connect 0x7fb4801457f0 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb46effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812012478 0 0) 0x7fb48013a690 con 0x7fb4801457f0 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb46effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb46effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb46effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb46effd640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:24:53.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.689+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb460004eb0 con 0x7fb4801457f0 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2177668604 0 0) 0x7fb460004eb0 con 0x7fb4801457f0 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb48013a690 con 0x7fb4801457f0 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb478001ec0 con 0x7fb4801457f0 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 10 monclient(hunting): dump: 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:53.681 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:53.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:53.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 10 client.?.objecter ms_handle_connect 0x7fb48013f650 2024-03-29T11:24:53.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:53.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 10 client.?.objecter ms_handle_connect 0x7fb4800a9d70 2024-03-29T11:24:53.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:53.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611217034 0 0) 0x7fb480071ce0 con 0x7fb48013f650 2024-03-29T11:24:53.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:53.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:53.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:53.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:24:53.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.690+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb460004eb0 con 0x7fb48013f650 2024-03-29T11:24:53.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1241036074 0 0) 0x7fb48013a690 con 0x7fb4801457f0 2024-03-29T11:24:53.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 >> v1:172.21.15.67:6790/0 conn(0x7fb48013f650 legacy=0x7fb480139870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:53.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 >> v1:172.21.15.67:6789/0 conn(0x7fb4800a9d70 legacy=0x7fb4801db620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb46effd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb46effd640 1 monclient: found mon.b 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb46effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb46effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb480070ab0 con 0x7fb4801457f0 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb46effd640 10 monclient: _finish_auth 0 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb46effd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb46effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb484bcd640 5 monclient: authenticate success, global_id 4387 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb484bcd640 10 monclient: _renew_subs 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb484bcd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb484bcd640 1 -- 172.21.15.67:0/1242323859 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48006f680 con 0x7fb4801457f0 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb484bcd640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb484bcd640 10 monclient: _renew_subs 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb484bcd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:24:53.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb484bcd640 1 -- 172.21.15.67:0/1242323859 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4801dbf10 con 0x7fb4801457f0 2024-03-29T11:24:53.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.691+0000 7fb484bcd640 1 librados: init done 2024-03-29T11:24:53.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.692+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb478003ce0 con 0x7fb4801457f0 2024-03-29T11:24:53.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.692+0000 7fb46effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:53.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.692+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb478002020 con 0x7fb4801457f0 2024-03-29T11:24:53.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.692+0000 7fb46effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:53.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.692+0000 7fb46effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:53.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.692+0000 7fb46effd640 10 monclient: dump: 2024-03-29T11:24:53.684 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:24:53.684 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:24:53.684 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:53.684 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:24:53.684 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:53.684 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:24:53.684 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:24:53.685 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:24:53.685 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:24:53.685 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:24:53.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.692+0000 7fb47cd99640 4 set_mon_vals no callback set 2024-03-29T11:24:53.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.692+0000 7fb484bcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:53.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.693+0000 7fb484bcd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:53.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.693+0000 7fb484bcd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:24:53.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.693+0000 7fb484bcd640 1 -- 172.21.15.67:0/1242323859 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb464005080 con 0x7fb4801457f0 2024-03-29T11:24:53.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.694+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fb4780040f0 con 0x7fb4801457f0 2024-03-29T11:24:53.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.694+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fb478085060 con 0x7fb4801457f0 2024-03-29T11:24:53.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.694+0000 7fb46effd640 10 client.4387.objecter ms_dispatch 0x7fb480000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:53.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.694+0000 7fb46effd640 3 client.4387.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:53.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.694+0000 7fb46effd640 3 client.4387.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:53.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.694+0000 7fb46effd640 20 client.4387.objecter dump_active .. 0 homeless 2024-03-29T11:24:53.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.694+0000 7fb47e59c640 10 monclient: get_auth_request con 0x7fb46006dd90 auth_method 0 2024-03-29T11:24:53.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.695+0000 7fb46effd640 10 client.4387.objecter ms_handle_connect 0x7fb46006dd90 2024-03-29T11:24:53.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.700+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb478050770 con 0x7fb4801457f0 2024-03-29T11:24:53.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.700+0000 7fb46effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:53.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:53.700+0000 7fb46effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:54.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.092+0000 7fb484bcd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.092+0000 7fb484bcd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.092+0000 7fb484bcd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:24:54.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.092+0000 7fb484bcd640 1 -- 172.21.15.67:0/1242323859 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb464005330 con 0x7fb4801457f0 2024-03-29T11:24:54.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.094+0000 7fb46effd640 1 -- 172.21.15.67:0/1242323859 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fb478052870 con 0x7fb4801457f0 2024-03-29T11:24:54.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.094+0000 7fb46effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.094+0000 7fb46effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:54.078 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:24:54.078 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:54.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.099+0000 7fb484bcd640 10 librados: watch_flush enter 2024-03-29T11:24:54.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.100+0000 7fb484bcd640 10 librados: watch_flush exit 2024-03-29T11:24:54.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.100+0000 7fb484bcd640 20 client.4387.objecter shutdown clearing up homeless session... 2024-03-29T11:24:54.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.100+0000 7fb484bcd640 10 client.4387.objecter successfully canceled tick 2024-03-29T11:24:54.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.100+0000 7fb484bcd640 1 -- 172.21.15.67:0/1242323859 >> v1:172.21.15.67:6816/404787197 conn(0x7fb46006dd90 legacy=0x7fb460070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.100+0000 7fb484bcd640 10 monclient: shutdown 2024-03-29T11:24:54.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.100+0000 7fb484bcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:54.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.100+0000 7fb484bcd640 1 -- 172.21.15.67:0/1242323859 >> v1:172.21.15.82:6789/0 conn(0x7fb4801457f0 legacy=0x7fb480139f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.100+0000 7fb484bcd640 1 -- 172.21.15.67:0/1242323859 shutdown_connections 2024-03-29T11:24:54.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.100+0000 7fb484bcd640 1 -- 172.21.15.67:0/1242323859 >> 172.21.15.67:0/1242323859 conn(0x7fb480066a00 msgr2=0x7fb480066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:54.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.101+0000 7fb484bcd640 1 -- 172.21.15.67:0/1242323859 shutdown_connections 2024-03-29T11:24:54.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.101+0000 7fb484bcd640 1 -- 172.21.15.67:0/1242323859 wait complete. 2024-03-29T11:24:54.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:24:54.101+0000 7fb484bcd640 1 librados: shutdown 2024-03-29T11:24:54.102 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:00.909 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:01.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.068+0000 7fa2bbe86640 10 monclient: get_monmap_and_config 2024-03-29T11:25:01.052 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.068+0000 7fa2bbe86640 10 monclient: build_initial_monmap 2024-03-29T11:25:01.052 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.068+0000 7fa2bbe86640 10 monclient: monmap: 2024-03-29T11:25:01.052 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:01.052 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:01.052 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:25:01.069453+0000 2024-03-29T11:25:01.052 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:25:01.069453+0000 2024-03-29T11:25:01.052 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:01.052 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:01.052 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:01.052 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:25:01.052 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:25:01.052 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:01.053 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.069+0000 7fa2bbe86640 1 Processor -- start 2024-03-29T11:25:01.067 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.070+0000 7fa2bbe86640 1 -- start start 2024-03-29T11:25:01.067 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.070+0000 7fa2bbe86640 10 monclient: init 2024-03-29T11:25:01.067 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.071+0000 7fa2bbe86640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:01.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.071+0000 7fa2bbe86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:01.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.071+0000 7fa2bbe86640 10 monclient(hunting): picked mon.noname-b con 0x7fa2b40b0770 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:01.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.071+0000 7fa2bbe86640 10 monclient(hunting): picked mon.noname-c con 0x7fa2b414a700 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:01.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.071+0000 7fa2bbe86640 10 monclient(hunting): picked mon.noname-a con 0x7fa2b40ac5a0 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:01.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.071+0000 7fa2bbe86640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b40aba70 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.071+0000 7fa2bbe86640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b40aecb0 con 0x7fa2b414a700 2024-03-29T11:25:01.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.071+0000 7fa2bbe86640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b40af150 con 0x7fa2b40b0770 2024-03-29T11:25:01.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.071+0000 7fa2bbe86640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:01.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.071+0000 7fa2bbe86640 10 monclient(hunting): authenticate will time out at 1161.367290s 2024-03-29T11:25:01.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.071+0000 7fa2ba423640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa2b40ac5a0 0x7fa2b4151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34872/0 (socket says 172.21.15.67:34872) 2024-03-29T11:25:01.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.071+0000 7fa2ba423640 1 -- 172.21.15.67:0/3202068178 learned_addr learned my addr 172.21.15.67:0/3202068178 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:01.068 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.072+0000 7fa2b8c20640 1 -- 172.21.15.67:0/3202068178 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232071946 0 0) 0x7fa2b40af150 con 0x7fa2b40b0770 2024-03-29T11:25:01.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.072+0000 7fa2b8c20640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.072+0000 7fa2b8c20640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.072+0000 7fa2b8c20640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.072+0000 7fa2b8c20640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:25:01.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.072+0000 7fa2b8c20640 1 -- 172.21.15.67:0/3202068178 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2a0003520 con 0x7fa2b40b0770 2024-03-29T11:25:01.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.072+0000 7fa2b8c20640 1 -- 172.21.15.67:0/3202068178 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081138999 0 0) 0x7fa2b40aba70 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.072+0000 7fa2b8c20640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.072+0000 7fa2b8c20640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.072+0000 7fa2b8c20640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.072+0000 7fa2b8c20640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:25:01.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.072+0000 7fa2b8c20640 1 -- 172.21.15.67:0/3202068178 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2b40af150 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 1 -- 172.21.15.67:0/3202068178 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3810256022 0 0) 0x7fa2a0003520 con 0x7fa2b40b0770 2024-03-29T11:25:01.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 1 -- 172.21.15.67:0/3202068178 >> v1:172.21.15.82:6789/0 conn(0x7fa2b414a700 legacy=0x7fa2b40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:01.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 1 -- 172.21.15.67:0/3202068178 >> v1:172.21.15.67:6789/0 conn(0x7fa2b40ac5a0 legacy=0x7fa2b4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 10 monclient: _finish_hunting 0 2024-03-29T11:25:01.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 1 monclient: found mon.noname-b 2024-03-29T11:25:01.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:01.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:25:01.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 1 -- 172.21.15.67:0/3202068178 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b40ac9d0 con 0x7fa2b40b0770 2024-03-29T11:25:01.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 10 monclient: _finish_auth 0 2024-03-29T11:25:01.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 10 monclient: _check_auth_tickets 2024-03-29T11:25:01.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:01.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2bbe86640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:25:01.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2bbe86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:01.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 1 -- 172.21.15.67:0/3202068178 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa2b0001690 con 0x7fa2b40b0770 2024-03-29T11:25:01.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.070 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:01.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 10 monclient: dump: 2024-03-29T11:25:01.071 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:01.071 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:01.071 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:01.071 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:01.071 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.071 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:01.071 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:01.071 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:01.071 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:01.071 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:01.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2bbe86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:01.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 1 -- 172.21.15.67:0/3202068178 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2b00018b0 con 0x7fa2b40b0770 2024-03-29T11:25:01.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2b8c20640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:01.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2bbe86640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:01.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.073+0000 7fa2bbe86640 4 set_mon_vals no callback set 2024-03-29T11:25:01.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.074+0000 7fa2bbe86640 10 monclient: shutdown 2024-03-29T11:25:01.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.074+0000 7fa2bbe86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:01.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.074+0000 7fa2bbe86640 1 -- 172.21.15.67:0/3202068178 >> v1:172.21.15.67:6790/0 conn(0x7fa2b40b0770 legacy=0x7fa2b4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.074+0000 7fa2bbe86640 1 -- 172.21.15.67:0/3202068178 shutdown_connections 2024-03-29T11:25:01.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.074+0000 7fa2bbe86640 1 -- 172.21.15.67:0/3202068178 >> 172.21.15.67:0/3202068178 conn(0x7fa2b406b660 msgr2=0x7fa2b406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:01.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.075+0000 7fa2bbe86640 1 -- 172.21.15.67:0/3202068178 shutdown_connections 2024-03-29T11:25:01.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.075+0000 7fa2bbe86640 1 -- 172.21.15.67:0/3202068178 wait complete. 2024-03-29T11:25:01.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.076+0000 7fa2bbe86640 10 monclient: build_initial_monmap 2024-03-29T11:25:01.072 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.076+0000 7fa2bbe86640 10 monclient: monmap: 2024-03-29T11:25:01.072 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:01.072 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:01.072 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:25:01.073 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:25:01.073 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:01.073 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:01.073 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:01.073 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:25:01.073 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:25:01.073 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:01.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.076+0000 7fa2bbe86640 1 librados: starting msgr at 2024-03-29T11:25:01.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.076+0000 7fa2bbe86640 1 librados: starting objecter 2024-03-29T11:25:01.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.076+0000 7fa2bbe86640 1 Processor -- start 2024-03-29T11:25:01.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.077+0000 7fa2bbe86640 1 -- start start 2024-03-29T11:25:01.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.077+0000 7fa2bbe86640 1 librados: setting wanted keys 2024-03-29T11:25:01.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.077+0000 7fa2bbe86640 1 librados: calling monclient init 2024-03-29T11:25:01.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.077+0000 7fa2bbe86640 10 monclient: init 2024-03-29T11:25:01.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2bbe86640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:01.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2bbe86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:01.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2bbe86640 10 monclient(hunting): picked mon.noname-a con 0x7fa2b40ac5a0 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:01.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2bbe86640 10 monclient(hunting): picked mon.noname-b con 0x7fa2b40b0770 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:01.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2bbe86640 10 monclient(hunting): picked mon.noname-c con 0x7fa2b414a700 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:01.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2bbe86640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b412a5b0 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2bbe86640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b4128280 con 0x7fa2b40b0770 2024-03-29T11:25:01.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2bbe86640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b4147cd0 con 0x7fa2b414a700 2024-03-29T11:25:01.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2bbe86640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:01.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2bbe86640 10 monclient(hunting): authenticate will time out at 1161.374031s 2024-03-29T11:25:01.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2b9c22640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa2b40ac5a0 0x7fa2b4146fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35080/0 (socket says 172.21.15.67:35080) 2024-03-29T11:25:01.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2b9c22640 1 -- 172.21.15.67:0/1706221023 learned_addr learned my addr 172.21.15.67:0/1706221023 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:01.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 10 client.?.objecter ms_handle_connect 0x7fa2b40ac5a0 2024-03-29T11:25:01.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:01.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 10 client.?.objecter ms_handle_connect 0x7fa2b414a700 2024-03-29T11:25:01.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:01.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 10 client.?.objecter ms_handle_connect 0x7fa2b40b0770 2024-03-29T11:25:01.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:01.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828775708 0 0) 0x7fa2b412a5b0 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:25:01.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa298004eb0 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64242891 0 0) 0x7fa2b4147cd0 con 0x7fa2b414a700 2024-03-29T11:25:01.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:25:01.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.078+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2b412a5b0 con 0x7fa2b414a700 2024-03-29T11:25:01.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227772955 0 0) 0x7fa2b4128280 con 0x7fa2b40b0770 2024-03-29T11:25:01.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:25:01.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2b4147cd0 con 0x7fa2b40b0770 2024-03-29T11:25:01.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 995266804 0 0) 0x7fa2b412a5b0 con 0x7fa2b414a700 2024-03-29T11:25:01.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2b4128280 con 0x7fa2b414a700 2024-03-29T11:25:01.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa2ac001b60 con 0x7fa2b414a700 2024-03-29T11:25:01.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.076 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:01.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient(hunting): dump: 2024-03-29T11:25:01.077 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:01.077 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:01.077 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:01.077 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:01.077 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.077 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:01.077 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:01.077 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:01.077 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:01.077 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:01.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:01.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2566496023 0 0) 0x7fa298004eb0 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.077 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2b412a5b0 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa2b0003790 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:01.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient(hunting): dump: 2024-03-29T11:25:01.078 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:01.078 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:01.078 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:01.078 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:01.078 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.078 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:01.078 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:01.078 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:01.078 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:01.078 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:01.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2052714806 0 0) 0x7fa2b4147cd0 con 0x7fa2b40b0770 2024-03-29T11:25:01.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa298004eb0 con 0x7fa2b40b0770 2024-03-29T11:25:01.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa2a4001640 con 0x7fa2b40b0770 2024-03-29T11:25:01.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:01.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient(hunting): dump: 2024-03-29T11:25:01.079 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:01.079 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:01.079 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:01.079 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:01.079 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.079 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:01.079 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:01.079 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:01.080 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:01.080 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:01.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 512574018 0 0) 0x7fa2b412a5b0 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 >> v1:172.21.15.67:6790/0 conn(0x7fa2b414a700 legacy=0x7fa2b4128eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 >> v1:172.21.15.82:6789/0 conn(0x7fa2b40b0770 legacy=0x7fa2b41287a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:01.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 monclient: found mon.a 2024-03-29T11:25:01.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:01.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:25:01.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b4127ec0 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:01.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:01.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:01.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2bbe86640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:25:01.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2bbe86640 10 monclient: _renew_subs 2024-03-29T11:25:01.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2bbe86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:25:01.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2bbe86640 1 -- 172.21.15.67:0/1706221023 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b41df6e0 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2bbe86640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:01.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2bbe86640 10 monclient: _renew_subs 2024-03-29T11:25:01.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2bbe86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:25:01.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2bbe86640 1 -- 172.21.15.67:0/1706221023 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b41deea0 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.079+0000 7fa2bbe86640 1 librados: init done 2024-03-29T11:25:01.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.080+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2b0001aa0 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.080+0000 7fa2aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:01.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.080+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa2b0003d80 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.080+0000 7fa2aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.080+0000 7fa2aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:01.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.080+0000 7fa2aa7fc640 10 monclient: dump: 2024-03-29T11:25:01.082 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:01.082 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:01.082 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:01.082 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:01.082 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.082 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:01.082 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:01.082 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:01.082 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:01.083 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:01.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.080+0000 7fa2abfff640 4 set_mon_vals no callback set 2024-03-29T11:25:01.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.082+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fa2b0003ee0 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.083+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fa2b0084ba0 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.083+0000 7fa2aa7fc640 10 client.4419.objecter ms_dispatch 0x7fa2b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:01.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.083+0000 7fa2aa7fc640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:01.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.083+0000 7fa2aa7fc640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:01.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.083+0000 7fa2aa7fc640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-29T11:25:01.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.084+0000 7fa2b9421640 10 monclient: get_auth_request con 0x7fa29806e1f0 auth_method 0 2024-03-29T11:25:01.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.084+0000 7fa2bbe86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.084+0000 7fa2aa7fc640 10 client.4419.objecter ms_handle_connect 0x7fa29806e1f0 2024-03-29T11:25:01.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.084+0000 7fa2bbe86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.084+0000 7fa2bbe86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:25:01.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.084+0000 7fa2bbe86640 1 -- 172.21.15.67:0/1706221023 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29c005080 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.095+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2b0050480 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.095+0000 7fa2aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.095+0000 7fa2aa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:01.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.461+0000 7fa2bbe86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:01.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.461+0000 7fa2bbe86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:01.444 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.461+0000 7fa2bbe86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:25:01.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.461+0000 7fa2bbe86640 1 -- 172.21.15.67:0/1706221023 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa29c005330 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.463+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/1706221023 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fa2b00523b0 con 0x7fa2b40ac5a0 2024-03-29T11:25:01.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.463+0000 7fa2aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:01.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.463+0000 7fa2aa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:01.446 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:25:01.446 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:01.450 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.466+0000 7fa2bbe86640 10 librados: watch_flush enter 2024-03-29T11:25:01.450 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.466+0000 7fa2bbe86640 10 librados: watch_flush exit 2024-03-29T11:25:01.450 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.466+0000 7fa2bbe86640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-29T11:25:01.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.466+0000 7fa2bbe86640 10 client.4419.objecter successfully canceled tick 2024-03-29T11:25:01.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.466+0000 7fa2bbe86640 1 -- 172.21.15.67:0/1706221023 >> v1:172.21.15.67:6816/404787197 conn(0x7fa29806e1f0 legacy=0x7fa2980706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.466+0000 7fa2bbe86640 10 monclient: shutdown 2024-03-29T11:25:01.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.466+0000 7fa2bbe86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:01.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.466+0000 7fa2bbe86640 1 -- 172.21.15.67:0/1706221023 >> v1:172.21.15.67:6789/0 conn(0x7fa2b40ac5a0 legacy=0x7fa2b4146fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.466+0000 7fa2bbe86640 1 -- 172.21.15.67:0/1706221023 shutdown_connections 2024-03-29T11:25:01.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.466+0000 7fa2bbe86640 1 -- 172.21.15.67:0/1706221023 >> 172.21.15.67:0/1706221023 conn(0x7fa2b406b660 msgr2=0x7fa2b406d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:01.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.467+0000 7fa2bbe86640 1 -- 172.21.15.67:0/1706221023 shutdown_connections 2024-03-29T11:25:01.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.467+0000 7fa2bbe86640 1 -- 172.21.15.67:0/1706221023 wait complete. 2024-03-29T11:25:01.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:01.467+0000 7fa2bbe86640 1 librados: shutdown 2024-03-29T11:25:01.464 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:08.272 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:08.416 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.432+0000 7ffac4c06640 10 monclient: get_monmap_and_config 2024-03-29T11:25:08.416 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.432+0000 7ffac4c06640 10 monclient: build_initial_monmap 2024-03-29T11:25:08.416 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.432+0000 7ffac4c06640 10 monclient: monmap: 2024-03-29T11:25:08.416 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:08.416 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:08.416 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:25:08.433271+0000 2024-03-29T11:25:08.416 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:25:08.433271+0000 2024-03-29T11:25:08.417 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:08.417 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:08.417 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:08.417 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:25:08.417 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:25:08.418 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:08.418 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.435+0000 7ffac4c06640 1 Processor -- start 2024-03-29T11:25:08.419 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.435+0000 7ffac4c06640 1 -- start start 2024-03-29T11:25:08.419 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.435+0000 7ffac4c06640 10 monclient: init 2024-03-29T11:25:08.421 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.436+0000 7ffac4c06640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:08.422 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.436+0000 7ffac4c06640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:08.422 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.436+0000 7ffac4c06640 10 monclient(hunting): picked mon.noname-b con 0x7ffac01457f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:08.422 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.437+0000 7ffac4c06640 10 monclient(hunting): picked mon.noname-a con 0x7ffac013f650 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:08.422 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.437+0000 7ffac4c06640 10 monclient(hunting): picked mon.noname-c con 0x7ffac00a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:08.422 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.437+0000 7ffac4c06640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac00aaef0 con 0x7ffac013f650 2024-03-29T11:25:08.422 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.437+0000 7ffac4c06640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac0145bd0 con 0x7ffac00a9d70 2024-03-29T11:25:08.422 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.437+0000 7ffac4c06640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac0145eb0 con 0x7ffac01457f0 2024-03-29T11:25:08.422 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.437+0000 7ffac4c06640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:08.422 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.437+0000 7ffac4c06640 10 monclient(hunting): authenticate will time out at 1168.733422s 2024-03-29T11:25:08.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.439+0000 7ffabdd9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffac013f650 0x7ffac00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41910/0 (socket says 172.21.15.67:41910) 2024-03-29T11:25:08.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.439+0000 7ffabdd9b640 1 -- 172.21.15.67:0/2444213661 learned_addr learned my addr 172.21.15.67:0/2444213661 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:08.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 1 -- 172.21.15.67:0/2444213661 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168647589 0 0) 0x7ffac00aaef0 con 0x7ffac013f650 2024-03-29T11:25:08.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:08.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:08.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:08.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:25:08.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 1 -- 172.21.15.67:0/2444213661 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa8003520 con 0x7ffac013f650 2024-03-29T11:25:08.428 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 1 -- 172.21.15.67:0/2444213661 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3778237915 0 0) 0x7ffaa8003520 con 0x7ffac013f650 2024-03-29T11:25:08.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 1 -- 172.21.15.67:0/2444213661 >> v1:172.21.15.67:6790/0 conn(0x7ffac01457f0 legacy=0x7ffac013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:08.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 1 -- 172.21.15.67:0/2444213661 >> v1:172.21.15.82:6789/0 conn(0x7ffac00a9d70 legacy=0x7ffac014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:08.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:25:08.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 1 monclient: found mon.noname-a 2024-03-29T11:25:08.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:08.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:25:08.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 1 -- 172.21.15.67:0/2444213661 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac0146340 con 0x7ffac013f650 2024-03-29T11:25:08.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 10 monclient: _finish_auth 0 2024-03-29T11:25:08.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:25:08.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:08.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 1 -- 172.21.15.67:0/2444213661 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ffab4001690 con 0x7ffac013f650 2024-03-29T11:25:08.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:08.429 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:08.430 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.440+0000 7ffabd59a640 10 monclient: dump: 2024-03-29T11:25:08.430 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:08.430 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:08.430 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:08.430 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:08.430 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:08.430 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:08.430 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:08.430 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:08.430 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:08.430 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:08.430 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.441+0000 7ffac4c06640 5 monclient: authenticate success, global_id 4422 2024-03-29T11:25:08.430 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.441+0000 7ffac4c06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:08.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.441+0000 7ffabd59a640 1 -- 172.21.15.67:0/2444213661 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffab4001860 con 0x7ffac013f650 2024-03-29T11:25:08.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.441+0000 7ffabd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:08.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.441+0000 7ffac4c06640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:08.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.441+0000 7ffac4c06640 4 set_mon_vals no callback set 2024-03-29T11:25:08.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.441+0000 7ffabd59a640 1 -- 172.21.15.67:0/2444213661 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ffab4001cb0 con 0x7ffac013f650 2024-03-29T11:25:08.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.441+0000 7ffabd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:08.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.441+0000 7ffabd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:08.431 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.441+0000 7ffabd59a640 10 monclient: dump: 2024-03-29T11:25:08.431 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:08.431 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:08.431 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:08.431 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:08.432 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:08.432 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:08.432 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:08.432 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:08.432 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:08.432 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:08.432 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.442+0000 7ffac4c06640 10 monclient: shutdown 2024-03-29T11:25:08.432 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.442+0000 7ffac4c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:08.432 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.442+0000 7ffac4c06640 1 -- 172.21.15.67:0/2444213661 >> v1:172.21.15.67:6789/0 conn(0x7ffac013f650 legacy=0x7ffac00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:08.432 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.442+0000 7ffac4c06640 1 -- 172.21.15.67:0/2444213661 shutdown_connections 2024-03-29T11:25:08.432 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.442+0000 7ffac4c06640 1 -- 172.21.15.67:0/2444213661 >> 172.21.15.67:0/2444213661 conn(0x7ffac0066a00 msgr2=0x7ffac0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:08.432 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.442+0000 7ffac4c06640 1 -- 172.21.15.67:0/2444213661 shutdown_connections 2024-03-29T11:25:08.432 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.442+0000 7ffac4c06640 1 -- 172.21.15.67:0/2444213661 wait complete. 2024-03-29T11:25:08.432 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.443+0000 7ffac4c06640 10 monclient: build_initial_monmap 2024-03-29T11:25:08.433 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.443+0000 7ffac4c06640 10 monclient: monmap: 2024-03-29T11:25:08.433 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:08.433 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:08.433 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:25:08.433 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:25:08.433 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:08.433 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:08.433 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:08.433 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:25:08.433 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:25:08.434 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:08.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.444+0000 7ffac4c06640 1 librados: starting msgr at 2024-03-29T11:25:08.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.444+0000 7ffac4c06640 1 librados: starting objecter 2024-03-29T11:25:08.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.444+0000 7ffac4c06640 1 Processor -- start 2024-03-29T11:25:08.434 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.444+0000 7ffac4c06640 1 -- start start 2024-03-29T11:25:08.435 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.444+0000 7ffac4c06640 1 librados: setting wanted keys 2024-03-29T11:25:08.437 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.444+0000 7ffac4c06640 1 librados: calling monclient init 2024-03-29T11:25:08.437 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.444+0000 7ffac4c06640 10 monclient: init 2024-03-29T11:25:08.438 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.445+0000 7ffac4c06640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:08.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.445+0000 7ffac4c06640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:08.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.445+0000 7ffac4c06640 10 monclient(hunting): picked mon.noname-b con 0x7ffac00a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:08.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.445+0000 7ffac4c06640 10 monclient(hunting): picked mon.noname-a con 0x7ffac013f650 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:08.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.445+0000 7ffac4c06640 10 monclient(hunting): picked mon.noname-c con 0x7ffac01457f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:08.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.445+0000 7ffac4c06640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac006f010 con 0x7ffac013f650 2024-03-29T11:25:08.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.445+0000 7ffac4c06640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac0142660 con 0x7ffac00a9d70 2024-03-29T11:25:08.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.445+0000 7ffac4c06640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac006f2f0 con 0x7ffac01457f0 2024-03-29T11:25:08.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.445+0000 7ffac4c06640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:08.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.445+0000 7ffac4c06640 10 monclient(hunting): authenticate will time out at 1168.741536s 2024-03-29T11:25:08.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.445+0000 7ffabdd9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffac013f650 0x7ffac0137450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42062/0 (socket says 172.21.15.67:42062) 2024-03-29T11:25:08.445 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.445+0000 7ffabdd9b640 1 -- 172.21.15.67:0/1741577962 learned_addr learned my addr 172.21.15.67:0/1741577962 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:08.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.446+0000 7ffaaeffd640 10 client.?.objecter ms_handle_connect 0x7ffac013f650 2024-03-29T11:25:08.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.446+0000 7ffaaeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:08.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.446+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242510862 0 0) 0x7ffac006f010 con 0x7ffac013f650 2024-03-29T11:25:08.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.446+0000 7ffaaeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:08.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.446+0000 7ffaaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:08.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.446+0000 7ffaaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:08.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.446+0000 7ffaaeffd640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:25:08.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.446+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa0004eb0 con 0x7ffac013f650 2024-03-29T11:25:08.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.449+0000 7ffaaeffd640 10 client.?.objecter ms_handle_connect 0x7ffac01457f0 2024-03-29T11:25:08.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.449+0000 7ffaaeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:08.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.449+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 842137765 0 0) 0x7ffaa0004eb0 con 0x7ffac013f650 2024-03-29T11:25:08.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.449+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffac006f010 con 0x7ffac013f650 2024-03-29T11:25:08.446 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.449+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ffab4001b70 con 0x7ffac013f650 2024-03-29T11:25:08.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.449+0000 7ffaaeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:08.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.449+0000 7ffaaeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:08.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.449+0000 7ffaaeffd640 10 monclient(hunting): dump: 2024-03-29T11:25:08.447 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:08.447 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:08.447 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:08.447 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:08.447 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:08.447 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:08.447 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:08.447 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:08.447 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:08.447 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:08.447 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:08.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759348706 0 0) 0x7ffac006f2f0 con 0x7ffac01457f0 2024-03-29T11:25:08.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:08.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:08.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:08.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:25:08.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa0004eb0 con 0x7ffac01457f0 2024-03-29T11:25:08.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1402003172 0 0) 0x7ffac006f010 con 0x7ffac013f650 2024-03-29T11:25:08.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 >> v1:172.21.15.67:6790/0 conn(0x7ffac01457f0 legacy=0x7ffac01e6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:08.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 >> v1:172.21.15.82:6789/0 conn(0x7ffac00a9d70 legacy=0x7ffac0141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:08.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:08.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 1 monclient: found mon.a 2024-03-29T11:25:08.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:08.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:25:08.448 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac0142c90 con 0x7ffac013f650 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffaaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffac4c06640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffac4c06640 10 monclient: _renew_subs 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffac4c06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.450+0000 7ffac4c06640 1 -- 172.21.15.67:0/1741577962 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac0142920 con 0x7ffac013f650 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.451+0000 7ffac4c06640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.451+0000 7ffac4c06640 10 monclient: _renew_subs 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.451+0000 7ffac4c06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.451+0000 7ffac4c06640 1 -- 172.21.15.67:0/1741577962 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac006e070 con 0x7ffac013f650 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.451+0000 7ffac4c06640 1 librados: init done 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.451+0000 7ffaaeffd640 10 client.4425.objecter ms_handle_connect 0x7ffac00a9d70 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.451+0000 7ffaaeffd640 10 client.4425.objecter resend_mon_ops 2024-03-29T11:25:08.449 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.451+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffab4001cd0 con 0x7ffac013f650 2024-03-29T11:25:08.450 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.451+0000 7ffaaeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:08.450 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.451+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ffab4004d90 con 0x7ffac013f650 2024-03-29T11:25:08.450 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.451+0000 7ffaaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:08.450 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.451+0000 7ffaaeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:08.450 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.451+0000 7ffaaeffd640 10 monclient: dump: 2024-03-29T11:25:08.450 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:08.450 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:08.450 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:08.450 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:08.450 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:08.450 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:08.450 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:08.450 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:08.450 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:08.451 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:08.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.451+0000 7ffabcd99640 4 set_mon_vals no callback set 2024-03-29T11:25:08.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.452+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7ffab401ad60 con 0x7ffac013f650 2024-03-29T11:25:08.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.453+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7ffab4084bd0 con 0x7ffac013f650 2024-03-29T11:25:08.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.453+0000 7ffaaeffd640 10 client.4425.objecter ms_dispatch 0x7ffac0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:08.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.453+0000 7ffaaeffd640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:08.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.453+0000 7ffaaeffd640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:08.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.453+0000 7ffaaeffd640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-29T11:25:08.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.454+0000 7ffabe59c640 10 monclient: get_auth_request con 0x7ffaa006dcb0 auth_method 0 2024-03-29T11:25:08.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.454+0000 7ffaaeffd640 10 client.4425.objecter ms_handle_connect 0x7ffaa006dcb0 2024-03-29T11:25:08.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.454+0000 7ffac4c06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:08.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.455+0000 7ffac4c06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:08.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.455+0000 7ffac4c06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:25:08.451 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.455+0000 7ffac4c06640 1 -- 172.21.15.67:0/1741577962 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa4005080 con 0x7ffac013f650 2024-03-29T11:25:08.452 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.465+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffab40504b0 con 0x7ffac013f650 2024-03-29T11:25:08.452 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.466+0000 7ffaaeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:08.452 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.466+0000 7ffaaeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:08.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.811+0000 7ffac4c06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:08.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.811+0000 7ffac4c06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:08.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.811+0000 7ffac4c06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:25:08.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.811+0000 7ffac4c06640 1 -- 172.21.15.67:0/1741577962 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffaa4005330 con 0x7ffac013f650 2024-03-29T11:25:08.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.812+0000 7ffaaeffd640 1 -- 172.21.15.67:0/1741577962 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7ffab40523e0 con 0x7ffac013f650 2024-03-29T11:25:08.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.812+0000 7ffaaeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:08.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.812+0000 7ffaaeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:08.795 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:25:08.795 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:08.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.815+0000 7ffac4c06640 10 librados: watch_flush enter 2024-03-29T11:25:08.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.815+0000 7ffac4c06640 10 librados: watch_flush exit 2024-03-29T11:25:08.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.815+0000 7ffac4c06640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-29T11:25:08.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.815+0000 7ffac4c06640 10 client.4425.objecter successfully canceled tick 2024-03-29T11:25:08.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.815+0000 7ffac4c06640 1 -- 172.21.15.67:0/1741577962 >> v1:172.21.15.67:6816/404787197 conn(0x7ffaa006dcb0 legacy=0x7ffaa0070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:08.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.815+0000 7ffac4c06640 10 monclient: shutdown 2024-03-29T11:25:08.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.815+0000 7ffac4c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:08.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.815+0000 7ffac4c06640 1 -- 172.21.15.67:0/1741577962 >> v1:172.21.15.67:6789/0 conn(0x7ffac013f650 legacy=0x7ffac0137450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:08.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.815+0000 7ffac4c06640 1 -- 172.21.15.67:0/1741577962 shutdown_connections 2024-03-29T11:25:08.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.815+0000 7ffac4c06640 1 -- 172.21.15.67:0/1741577962 >> 172.21.15.67:0/1741577962 conn(0x7ffac0066a00 msgr2=0x7ffac013b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:08.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.816+0000 7ffac4c06640 1 -- 172.21.15.67:0/1741577962 shutdown_connections 2024-03-29T11:25:08.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.816+0000 7ffac4c06640 1 -- 172.21.15.67:0/1741577962 wait complete. 2024-03-29T11:25:08.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:08.816+0000 7ffac4c06640 1 librados: shutdown 2024-03-29T11:25:08.816 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:15.624 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:15.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.789+0000 7fd23d886640 10 monclient: get_monmap_and_config 2024-03-29T11:25:15.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.789+0000 7fd23d886640 10 monclient: build_initial_monmap 2024-03-29T11:25:15.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.789+0000 7fd23d886640 10 monclient: monmap: 2024-03-29T11:25:15.774 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:15.774 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:15.775 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:25:15.789899+0000 2024-03-29T11:25:15.775 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:25:15.789899+0000 2024-03-29T11:25:15.775 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:15.775 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:15.775 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:15.775 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:25:15.776 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:25:15.776 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:15.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.790+0000 7fd23d886640 1 Processor -- start 2024-03-29T11:25:15.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.790+0000 7fd23d886640 1 -- start start 2024-03-29T11:25:15.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.790+0000 7fd23d886640 10 monclient: init 2024-03-29T11:25:15.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.791+0000 7fd23d886640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:15.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.791+0000 7fd23d886640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:15.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.791+0000 7fd23d886640 10 monclient(hunting): picked mon.noname-b con 0x7fd2380acf60 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:15.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.791+0000 7fd23d886640 10 monclient(hunting): picked mon.noname-a con 0x7fd23813ff40 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:15.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.791+0000 7fd23d886640 10 monclient(hunting): picked mon.noname-c con 0x7fd2381403f0 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:15.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.791+0000 7fd23d886640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2380b4310 con 0x7fd23813ff40 2024-03-29T11:25:15.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.791+0000 7fd23d886640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd238140800 con 0x7fd2381403f0 2024-03-29T11:25:15.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.791+0000 7fd23d886640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2380aec00 con 0x7fd2380acf60 2024-03-29T11:25:15.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.791+0000 7fd23d886640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:15.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.791+0000 7fd23d886640 10 monclient(hunting): authenticate will time out at 1176.087510s 2024-03-29T11:25:15.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.794+0000 7fd236ffd640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fd2380acf60 0x7fd238139b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:52456/0 (socket says 172.21.15.67:52456) 2024-03-29T11:25:15.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.794+0000 7fd236ffd640 1 -- 172.21.15.67:0/449533910 learned_addr learned my addr 172.21.15.67:0/449533910 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:15.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.795+0000 7fd235ffb640 1 -- 172.21.15.67:0/449533910 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903805298 0 0) 0x7fd2380aec00 con 0x7fd2380acf60 2024-03-29T11:25:15.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.795+0000 7fd235ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:15.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.795+0000 7fd235ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:15.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.795+0000 7fd235ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:15.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.795+0000 7fd235ffb640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:25:15.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.795+0000 7fd235ffb640 1 -- 172.21.15.67:0/449533910 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd220003520 con 0x7fd2380acf60 2024-03-29T11:25:15.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.795+0000 7fd235ffb640 1 -- 172.21.15.67:0/449533910 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743980466 0 0) 0x7fd2380b4310 con 0x7fd23813ff40 2024-03-29T11:25:15.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.795+0000 7fd235ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:15.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.795+0000 7fd235ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:15.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.795+0000 7fd235ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:15.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.795+0000 7fd235ffb640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:25:15.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.795+0000 7fd235ffb640 1 -- 172.21.15.67:0/449533910 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2380aec00 con 0x7fd23813ff40 2024-03-29T11:25:15.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.795+0000 7fd235ffb640 1 -- 172.21.15.67:0/449533910 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1428628986 0 0) 0x7fd2380aec00 con 0x7fd23813ff40 2024-03-29T11:25:15.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 1 -- 172.21.15.67:0/449533910 >> v1:172.21.15.67:6790/0 conn(0x7fd2380acf60 legacy=0x7fd238139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:15.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 1 -- 172.21.15.67:0/449533910 >> v1:172.21.15.82:6789/0 conn(0x7fd2381403f0 legacy=0x7fd23814bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:15.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:15.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 1 monclient: found mon.noname-a 2024-03-29T11:25:15.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:15.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:25:15.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 1 -- 172.21.15.67:0/449533910 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2380aeee0 con 0x7fd23813ff40 2024-03-29T11:25:15.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:15.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:15.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:15.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 1 -- 172.21.15.67:0/449533910 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd22c001670 con 0x7fd23813ff40 2024-03-29T11:25:15.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:15.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:15.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 10 monclient: dump: 2024-03-29T11:25:15.791 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:15.791 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:15.791 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:15.791 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:15.791 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:15.791 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:15.792 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:15.792 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:15.792 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:15.792 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:15.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 1 -- 172.21.15.67:0/449533910 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd22c001890 con 0x7fd23813ff40 2024-03-29T11:25:15.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:15.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 1 -- 172.21.15.67:0/449533910 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd22c001c20 con 0x7fd23813ff40 2024-03-29T11:25:15.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:15.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:15.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd235ffb640 10 monclient: dump: 2024-03-29T11:25:15.793 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:15.793 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:15.793 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:15.793 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:15.793 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:15.793 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:15.793 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:15.793 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:15.793 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:15.794 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:15.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd23d886640 5 monclient: authenticate success, global_id 4428 2024-03-29T11:25:15.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd23d886640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:15.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.796+0000 7fd23d886640 4 set_mon_vals no callback set 2024-03-29T11:25:15.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.797+0000 7fd23d886640 10 monclient: shutdown 2024-03-29T11:25:15.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.797+0000 7fd23d886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:15.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.797+0000 7fd23d886640 1 -- 172.21.15.67:0/449533910 >> v1:172.21.15.67:6789/0 conn(0x7fd23813ff40 legacy=0x7fd238148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:15.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.797+0000 7fd23d886640 1 -- 172.21.15.67:0/449533910 shutdown_connections 2024-03-29T11:25:15.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.797+0000 7fd23d886640 1 -- 172.21.15.67:0/449533910 >> 172.21.15.67:0/449533910 conn(0x7fd2380b35e0 msgr2=0x7fd2380b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:15.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.798+0000 7fd23d886640 1 -- 172.21.15.67:0/449533910 shutdown_connections 2024-03-29T11:25:15.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.799+0000 7fd23d886640 1 -- 172.21.15.67:0/449533910 wait complete. 2024-03-29T11:25:15.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.800+0000 7fd23d886640 10 monclient: build_initial_monmap 2024-03-29T11:25:15.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.800+0000 7fd23d886640 10 monclient: monmap: 2024-03-29T11:25:15.795 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:15.795 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:15.795 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:25:15.795 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:25:15.795 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:15.795 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:15.795 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:15.795 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:25:15.795 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:25:15.795 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:15.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.801+0000 7fd23d886640 1 librados: starting msgr at 2024-03-29T11:25:15.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.801+0000 7fd23d886640 1 librados: starting objecter 2024-03-29T11:25:15.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.801+0000 7fd23d886640 1 Processor -- start 2024-03-29T11:25:15.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.801+0000 7fd23d886640 1 -- start start 2024-03-29T11:25:15.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.801+0000 7fd23d886640 1 librados: setting wanted keys 2024-03-29T11:25:15.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.801+0000 7fd23d886640 1 librados: calling monclient init 2024-03-29T11:25:15.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.801+0000 7fd23d886640 10 monclient: init 2024-03-29T11:25:15.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.802+0000 7fd23d886640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:15.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.802+0000 7fd23d886640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:15.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.802+0000 7fd23d886640 10 monclient(hunting): picked mon.noname-a con 0x7fd2380acf60 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:15.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.802+0000 7fd23d886640 10 monclient(hunting): picked mon.noname-b con 0x7fd23813ff40 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:15.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.804+0000 7fd23d886640 10 monclient(hunting): picked mon.noname-c con 0x7fd2381403f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:15.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.804+0000 7fd23d886640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2381ca3f0 con 0x7fd2380acf60 2024-03-29T11:25:15.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.804+0000 7fd23d886640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd23813afb0 con 0x7fd23813ff40 2024-03-29T11:25:15.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.804+0000 7fd23d886640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd23813b290 con 0x7fd2381403f0 2024-03-29T11:25:15.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.804+0000 7fd23d886640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:15.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.804+0000 7fd23d886640 10 monclient(hunting): authenticate will time out at 1176.099987s 2024-03-29T11:25:15.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.804+0000 7fd2377fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fd2381403f0 0x7fd2381e6240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:53750/0 (socket says 172.21.15.67:53750) 2024-03-29T11:25:15.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.804+0000 7fd2377fe640 1 -- 172.21.15.67:0/2989152122 learned_addr learned my addr 172.21.15.67:0/2989152122 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:15.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.804+0000 7fd21ffff640 10 client.?.objecter ms_handle_connect 0x7fd2381403f0 2024-03-29T11:25:15.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.804+0000 7fd21ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:15.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.805+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985984198 0 0) 0x7fd23813b290 con 0x7fd2381403f0 2024-03-29T11:25:15.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.805+0000 7fd21ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:15.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.805+0000 7fd21ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:15.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.805+0000 7fd21ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:15.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.805+0000 7fd21ffff640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:25:15.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.805+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd224004eb0 con 0x7fd2381403f0 2024-03-29T11:25:15.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.805+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2830060499 0 0) 0x7fd224004eb0 con 0x7fd2381403f0 2024-03-29T11:25:15.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.805+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd23813b290 con 0x7fd2381403f0 2024-03-29T11:25:15.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.805+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd230001c90 con 0x7fd2381403f0 2024-03-29T11:25:15.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.805+0000 7fd21ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:15.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.805+0000 7fd21ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:15.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.805+0000 7fd21ffff640 10 monclient(hunting): dump: 2024-03-29T11:25:15.800 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:15.800 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:15.801 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:15.801 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:15.801 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:15.801 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:15.801 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:15.801 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:15.801 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:15.801 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:15.801 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.805+0000 7fd21ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:15.801 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 735833128 0 0) 0x7fd23813b290 con 0x7fd2381403f0 2024-03-29T11:25:15.801 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 >> v1:172.21.15.82:6789/0 conn(0x7fd23813ff40 legacy=0x7fd2380ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:15.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 >> v1:172.21.15.67:6789/0 conn(0x7fd2380acf60 legacy=0x7fd2380aad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:15.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:15.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 1 monclient: found mon.c 2024-03-29T11:25:15.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:15.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:15.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23813b570 con 0x7fd2381403f0 2024-03-29T11:25:15.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 10 monclient: _finish_auth 0 2024-03-29T11:25:15.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:15.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:15.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd23d886640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:25:15.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd23d886640 10 monclient: _renew_subs 2024-03-29T11:25:15.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd23d886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:15.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd23d886640 1 -- 172.21.15.67:0/2989152122 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2380ac960 con 0x7fd2381403f0 2024-03-29T11:25:15.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd230003990 con 0x7fd2381403f0 2024-03-29T11:25:15.803 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:15.803 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd230003da0 con 0x7fd2381403f0 2024-03-29T11:25:15.803 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:15.803 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:15.803 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd21ffff640 10 monclient: dump: 2024-03-29T11:25:15.803 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:15.803 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:15.803 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:15.803 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:15.803 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:15.803 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:15.803 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:15.803 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:15.803 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:15.804 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:15.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd234ff9640 4 set_mon_vals no callback set 2024-03-29T11:25:15.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd23d886640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:15.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd23d886640 10 monclient: _renew_subs 2024-03-29T11:25:15.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd23d886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:15.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd23d886640 1 -- 172.21.15.67:0/2989152122 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2381c9a60 con 0x7fd2381403f0 2024-03-29T11:25:15.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.806+0000 7fd23d886640 1 librados: init done 2024-03-29T11:25:15.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.809+0000 7fd23d886640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:15.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.809+0000 7fd23d886640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:15.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.809+0000 7fd23d886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:15.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.809+0000 7fd23d886640 1 -- 172.21.15.67:0/2989152122 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd238075f10 con 0x7fd2381403f0 2024-03-29T11:25:15.805 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.809+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fd230003f00 con 0x7fd2381403f0 2024-03-29T11:25:15.805 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.810+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fd230054c60 con 0x7fd2381403f0 2024-03-29T11:25:15.805 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.810+0000 7fd21ffff640 10 client.4469.objecter ms_dispatch 0x7fd238000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:15.805 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.810+0000 7fd21ffff640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:15.805 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.810+0000 7fd21ffff640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:15.805 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.810+0000 7fd21ffff640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-29T11:25:15.805 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.810+0000 7fd236ffd640 10 monclient: get_auth_request con 0x7fd22406dbc0 auth_method 0 2024-03-29T11:25:15.805 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.811+0000 7fd21ffff640 10 client.4469.objecter ms_handle_connect 0x7fd22406dbc0 2024-03-29T11:25:15.806 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.819+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd230050f00 con 0x7fd2381403f0 2024-03-29T11:25:15.806 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.819+0000 7fd21ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:15.806 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:15.819+0000 7fd21ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:16.140 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.157+0000 7fd23d886640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.140 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.157+0000 7fd23d886640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.157+0000 7fd23d886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.157+0000 7fd23d886640 1 -- 172.21.15.67:0/2989152122 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2380aa9e0 con 0x7fd2381403f0 2024-03-29T11:25:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.158+0000 7fd21ffff640 1 -- 172.21.15.67:0/2989152122 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fd2300510a0 con 0x7fd2381403f0 2024-03-29T11:25:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.158+0000 7fd21ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.158+0000 7fd21ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:16.143 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:25:16.143 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:16.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.162+0000 7fd23d886640 10 librados: watch_flush enter 2024-03-29T11:25:16.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.163+0000 7fd23d886640 10 librados: watch_flush exit 2024-03-29T11:25:16.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.163+0000 7fd23d886640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-29T11:25:16.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.163+0000 7fd23d886640 10 client.4469.objecter successfully canceled tick 2024-03-29T11:25:16.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.163+0000 7fd23d886640 1 -- 172.21.15.67:0/2989152122 >> v1:172.21.15.67:6816/404787197 conn(0x7fd22406dbc0 legacy=0x7fd224070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.163+0000 7fd23d886640 10 monclient: shutdown 2024-03-29T11:25:16.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.163+0000 7fd23d886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:16.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.163+0000 7fd23d886640 1 -- 172.21.15.67:0/2989152122 >> v1:172.21.15.67:6790/0 conn(0x7fd2381403f0 legacy=0x7fd2381e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.163+0000 7fd23d886640 1 -- 172.21.15.67:0/2989152122 shutdown_connections 2024-03-29T11:25:16.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.163+0000 7fd23d886640 1 -- 172.21.15.67:0/2989152122 >> 172.21.15.67:0/2989152122 conn(0x7fd2380b35e0 msgr2=0x7fd2380b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:16.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.163+0000 7fd23d886640 1 -- 172.21.15.67:0/2989152122 shutdown_connections 2024-03-29T11:25:16.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.163+0000 7fd23d886640 1 -- 172.21.15.67:0/2989152122 wait complete. 2024-03-29T11:25:16.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:16.163+0000 7fd23d886640 1 librados: shutdown 2024-03-29T11:25:16.166 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:22.972 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:23.119 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.136+0000 7f8feceba640 10 monclient: get_monmap_and_config 2024-03-29T11:25:23.120 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.136+0000 7f8feceba640 10 monclient: build_initial_monmap 2024-03-29T11:25:23.120 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.136+0000 7f8feceba640 10 monclient: monmap: 2024-03-29T11:25:23.120 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:23.120 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:23.120 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:25:23.137091+0000 2024-03-29T11:25:23.120 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:25:23.137091+0000 2024-03-29T11:25:23.120 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:23.120 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:23.120 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:23.120 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:25:23.120 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:25:23.121 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:23.121 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.137+0000 7f8feceba640 1 Processor -- start 2024-03-29T11:25:23.121 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.137+0000 7f8feceba640 1 -- start start 2024-03-29T11:25:23.121 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.137+0000 7f8feceba640 10 monclient: init 2024-03-29T11:25:23.121 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.138+0000 7f8feceba640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:23.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.138+0000 7f8feceba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:23.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.138+0000 7f8feceba640 10 monclient(hunting): picked mon.noname-a con 0x7f8fe80b1f60 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:23.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.138+0000 7f8feceba640 10 monclient(hunting): picked mon.noname-b con 0x7f8fe8071eb0 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:23.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.138+0000 7f8feceba640 10 monclient(hunting): picked mon.noname-c con 0x7f8fe8143d10 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:23.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.138+0000 7f8feceba640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe806ef80 con 0x7f8fe80b1f60 2024-03-29T11:25:23.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.138+0000 7f8feceba640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe806f260 con 0x7f8fe8143d10 2024-03-29T11:25:23.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.138+0000 7f8feceba640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe80b40c0 con 0x7f8fe8071eb0 2024-03-29T11:25:23.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.138+0000 7f8feceba640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:23.123 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.138+0000 7f8feceba640 10 monclient(hunting): authenticate will time out at 1183.434350s 2024-03-29T11:25:23.124 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.141+0000 7f8fdffff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f8fe8071eb0 0x7f8fe8146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:51448/0 (socket says 172.21.15.67:51448) 2024-03-29T11:25:23.125 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.141+0000 7f8fdffff640 1 -- 172.21.15.67:0/505840939 learned_addr learned my addr 172.21.15.67:0/505840939 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:23.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.142+0000 7f8fe5d9b640 1 -- 172.21.15.67:0/505840939 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841630034 0 0) 0x7f8fe806ef80 con 0x7f8fe80b1f60 2024-03-29T11:25:23.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.142+0000 7f8fe5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.142+0000 7f8fe5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.142+0000 7f8fe5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.142+0000 7f8fe5d9b640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:25:23.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.142+0000 7f8fe5d9b640 1 -- 172.21.15.67:0/505840939 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd4003520 con 0x7f8fe80b1f60 2024-03-29T11:25:23.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.142+0000 7f8fe5d9b640 1 -- 172.21.15.67:0/505840939 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3895420789 0 0) 0x7f8fd4003520 con 0x7f8fe80b1f60 2024-03-29T11:25:23.129 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.142+0000 7f8fe5d9b640 1 -- 172.21.15.67:0/505840939 >> v1:172.21.15.67:6790/0 conn(0x7f8fe8071eb0 legacy=0x7f8fe8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:23.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.142+0000 7f8fe5d9b640 1 -- 172.21.15.67:0/505840939 >> v1:172.21.15.82:6789/0 conn(0x7f8fe8143d10 legacy=0x7f8fe8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.142+0000 7f8fe5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:25:23.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.142+0000 7f8fe5d9b640 1 monclient: found mon.noname-a 2024-03-29T11:25:23.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.142+0000 7f8fe5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:23.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.142+0000 7f8fe5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:25:23.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.142+0000 7f8fe5d9b640 1 -- 172.21.15.67:0/505840939 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe8155d70 con 0x7f8fe80b1f60 2024-03-29T11:25:23.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8fe5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:25:23.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8fe5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:25:23.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8fe5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:23.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8feceba640 5 monclient: authenticate success, global_id 4431 2024-03-29T11:25:23.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8feceba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:23.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8fe5d9b640 1 -- 172.21.15.67:0/505840939 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f8fd8001690 con 0x7f8fe80b1f60 2024-03-29T11:25:23.130 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8fe5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8fe5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8fe5d9b640 10 monclient: dump: 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8fe5d9b640 1 -- 172.21.15.67:0/505840939 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fd8001920 con 0x7f8fe80b1f60 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8fe5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:23.131 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8fe5d9b640 1 -- 172.21.15.67:0/505840939 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f8fd8001cb0 con 0x7f8fe80b1f60 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8fe5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8fe5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8fe5d9b640 10 monclient: dump: 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8feceba640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:23.132 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.143+0000 7f8feceba640 4 set_mon_vals no callback set 2024-03-29T11:25:23.133 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.145+0000 7f8feceba640 10 monclient: shutdown 2024-03-29T11:25:23.133 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.145+0000 7f8feceba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:23.133 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.145+0000 7f8feceba640 1 -- 172.21.15.67:0/505840939 >> v1:172.21.15.67:6789/0 conn(0x7f8fe80b1f60 legacy=0x7f8fe8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.134 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.146+0000 7f8feceba640 1 -- 172.21.15.67:0/505840939 shutdown_connections 2024-03-29T11:25:23.135 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.146+0000 7f8feceba640 1 -- 172.21.15.67:0/505840939 >> 172.21.15.67:0/505840939 conn(0x7f8fe8066a00 msgr2=0x7f8fe8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:23.135 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.147+0000 7f8feceba640 1 -- 172.21.15.67:0/505840939 shutdown_connections 2024-03-29T11:25:23.135 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.147+0000 7f8feceba640 1 -- 172.21.15.67:0/505840939 wait complete. 2024-03-29T11:25:23.135 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.149+0000 7f8feceba640 10 monclient: build_initial_monmap 2024-03-29T11:25:23.135 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.149+0000 7f8feceba640 10 monclient: monmap: 2024-03-29T11:25:23.135 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:23.135 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:23.135 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:25:23.135 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:25:23.135 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:23.135 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:23.135 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:23.135 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:25:23.136 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:25:23.136 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:23.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.149+0000 7f8feceba640 1 librados: starting msgr at 2024-03-29T11:25:23.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.149+0000 7f8feceba640 1 librados: starting objecter 2024-03-29T11:25:23.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.149+0000 7f8feceba640 1 Processor -- start 2024-03-29T11:25:23.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.149+0000 7f8feceba640 1 -- start start 2024-03-29T11:25:23.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.149+0000 7f8feceba640 1 librados: setting wanted keys 2024-03-29T11:25:23.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.149+0000 7f8feceba640 1 librados: calling monclient init 2024-03-29T11:25:23.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.149+0000 7f8feceba640 10 monclient: init 2024-03-29T11:25:23.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.150+0000 7f8feceba640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:23.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.150+0000 7f8feceba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:23.136 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.150+0000 7f8feceba640 10 monclient(hunting): picked mon.noname-c con 0x7f8fe80b1f60 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:23.141 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.152+0000 7f8feceba640 10 monclient(hunting): picked mon.noname-a con 0x7f8fe8143d10 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:23.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.152+0000 7f8feceba640 10 monclient(hunting): picked mon.noname-b con 0x7f8fe8154a20 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:23.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.152+0000 7f8feceba640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe8154e40 con 0x7f8fe8143d10 2024-03-29T11:25:23.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.152+0000 7f8feceba640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe814b260 con 0x7f8fe8154a20 2024-03-29T11:25:23.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.152+0000 7f8feceba640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe814b540 con 0x7f8fe80b1f60 2024-03-29T11:25:23.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.152+0000 7f8feceba640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:23.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.152+0000 7f8feceba640 10 monclient(hunting): authenticate will time out at 1183.448729s 2024-03-29T11:25:23.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.152+0000 7f8fe659c640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f8fe80b1f60 0x7f8fe813fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:52866/0 (socket says 172.21.15.67:52866) 2024-03-29T11:25:23.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.152+0000 7f8fe659c640 1 -- 172.21.15.67:0/4111371497 learned_addr learned my addr 172.21.15.67:0/4111371497 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:23.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.153+0000 7f8fdeffd640 10 client.?.objecter ms_handle_connect 0x7f8fe80b1f60 2024-03-29T11:25:23.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.153+0000 7f8fdeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:23.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.156+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576858810 0 0) 0x7f8fe814b540 con 0x7f8fe80b1f60 2024-03-29T11:25:23.142 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.156+0000 7f8fdeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.156+0000 7f8fdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.156+0000 7f8fdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.156+0000 7f8fdeffd640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:25:23.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.156+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc8004eb0 con 0x7f8fe80b1f60 2024-03-29T11:25:23.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.156+0000 7f8fdeffd640 10 client.?.objecter ms_handle_connect 0x7f8fe8143d10 2024-03-29T11:25:23.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.156+0000 7f8fdeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:23.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.156+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 295910688 0 0) 0x7f8fc8004eb0 con 0x7f8fe80b1f60 2024-03-29T11:25:23.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.156+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fe814b540 con 0x7f8fe80b1f60 2024-03-29T11:25:23.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.156+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f8fd8001650 con 0x7f8fe80b1f60 2024-03-29T11:25:23.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.156+0000 7f8fdeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.156+0000 7f8fdeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:23.143 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.156+0000 7f8fdeffd640 10 monclient(hunting): dump: 2024-03-29T11:25:23.143 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:23.144 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:23.144 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:23.144 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:23.144 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.144 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:23.144 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:23.144 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:23.144 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:23.144 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:23.144 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:23.144 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454141783 0 0) 0x7f8fe8154e40 con 0x7f8fe8143d10 2024-03-29T11:25:23.144 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.144 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.144 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:25:23.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc8004eb0 con 0x7f8fe8143d10 2024-03-29T11:25:23.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3482764735 0 0) 0x7f8fc8004eb0 con 0x7f8fe8143d10 2024-03-29T11:25:23.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fe8154e40 con 0x7f8fe8143d10 2024-03-29T11:25:23.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f8fd00014d0 con 0x7f8fe8143d10 2024-03-29T11:25:23.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:23.145 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 10 monclient(hunting): dump: 2024-03-29T11:25:23.145 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:23.145 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:23.145 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:23.145 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:23.145 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.145 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:23.146 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:23.146 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:23.146 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:23.146 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:23.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1363462135 0 0) 0x7f8fe814b540 con 0x7f8fe80b1f60 2024-03-29T11:25:23.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 >> v1:172.21.15.82:6789/0 conn(0x7f8fe8154a20 legacy=0x7f8fe81def50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 >> v1:172.21.15.67:6789/0 conn(0x7f8fe8143d10 legacy=0x7f8fe8151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:23.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 1 monclient: found mon.c 2024-03-29T11:25:23.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:23.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:23.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe81405a0 con 0x7f8fe80b1f60 2024-03-29T11:25:23.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:23.146 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8fdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8feceba640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8feceba640 10 monclient: _renew_subs 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8feceba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8feceba640 1 -- 172.21.15.67:0/4111371497 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe81526e0 con 0x7f8fe80b1f60 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8feceba640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8feceba640 10 monclient: _renew_subs 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8feceba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8feceba640 1 -- 172.21.15.67:0/4111371497 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe8152b70 con 0x7f8fe80b1f60 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.157+0000 7f8feceba640 1 librados: init done 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.158+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fd8001270 con 0x7f8fe80b1f60 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.158+0000 7f8fdeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.158+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f8fd8003b00 con 0x7f8fe80b1f60 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.158+0000 7f8fdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.158+0000 7f8fdeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:23.148 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.158+0000 7f8fdeffd640 10 monclient: dump: 2024-03-29T11:25:23.148 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:23.148 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:23.148 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:23.148 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:23.148 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.148 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:23.148 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:23.148 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:23.148 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:23.148 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:23.149 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.158+0000 7f8fe559a640 4 set_mon_vals no callback set 2024-03-29T11:25:23.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.159+0000 7f8feceba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.159+0000 7f8feceba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.159+0000 7f8feceba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:23.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.159+0000 7f8feceba640 1 -- 172.21.15.67:0/4111371497 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc005080 con 0x7f8fe80b1f60 2024-03-29T11:25:23.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.160+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f8fd8003c60 con 0x7f8fe80b1f60 2024-03-29T11:25:23.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.161+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f8fd8084b80 con 0x7f8fe80b1f60 2024-03-29T11:25:23.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.161+0000 7f8fdeffd640 10 client.4472.objecter ms_dispatch 0x7f8fe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:23.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.161+0000 7f8fdeffd640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:23.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.161+0000 7f8fdeffd640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:23.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.161+0000 7f8fdeffd640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-29T11:25:23.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.161+0000 7f8fdffff640 10 monclient: get_auth_request con 0x7f8fc806ddb0 auth_method 0 2024-03-29T11:25:23.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.161+0000 7f8fdeffd640 10 client.4472.objecter ms_handle_connect 0x7f8fc806ddb0 2024-03-29T11:25:23.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.169+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8fd8050290 con 0x7f8fe80b1f60 2024-03-29T11:25:23.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.169+0000 7f8fdeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.169+0000 7f8fdeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:23.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.509+0000 7f8feceba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:23.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.509+0000 7f8feceba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:23.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.509+0000 7f8feceba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:23.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.509+0000 7f8feceba640 1 -- 172.21.15.67:0/4111371497 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fcc005330 con 0x7f8fe80b1f60 2024-03-29T11:25:23.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.512+0000 7f8fdeffd640 1 -- 172.21.15.67:0/4111371497 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f8fd8052390 con 0x7f8fe80b1f60 2024-03-29T11:25:23.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.512+0000 7f8fdeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:23.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.512+0000 7f8fdeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:23.497 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:25:23.497 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:23.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.516+0000 7f8feceba640 10 librados: watch_flush enter 2024-03-29T11:25:23.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.516+0000 7f8feceba640 10 librados: watch_flush exit 2024-03-29T11:25:23.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.516+0000 7f8feceba640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-29T11:25:23.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.516+0000 7f8feceba640 10 client.4472.objecter successfully canceled tick 2024-03-29T11:25:23.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.517+0000 7f8feceba640 1 -- 172.21.15.67:0/4111371497 >> v1:172.21.15.67:6816/404787197 conn(0x7f8fc806ddb0 legacy=0x7f8fc8070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.517+0000 7f8feceba640 10 monclient: shutdown 2024-03-29T11:25:23.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.517+0000 7f8feceba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:23.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.517+0000 7f8feceba640 1 -- 172.21.15.67:0/4111371497 >> v1:172.21.15.67:6790/0 conn(0x7f8fe80b1f60 legacy=0x7f8fe813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.517+0000 7f8feceba640 1 -- 172.21.15.67:0/4111371497 shutdown_connections 2024-03-29T11:25:23.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.517+0000 7f8feceba640 1 -- 172.21.15.67:0/4111371497 >> 172.21.15.67:0/4111371497 conn(0x7f8fe8066a00 msgr2=0x7f8fe8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:23.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.517+0000 7f8feceba640 1 -- 172.21.15.67:0/4111371497 shutdown_connections 2024-03-29T11:25:23.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.517+0000 7f8feceba640 1 -- 172.21.15.67:0/4111371497 wait complete. 2024-03-29T11:25:23.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:23.517+0000 7f8feceba640 1 librados: shutdown 2024-03-29T11:25:23.520 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:30.327 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:30.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.498+0000 7f876134b640 10 monclient: get_monmap_and_config 2024-03-29T11:25:30.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.498+0000 7f876134b640 10 monclient: build_initial_monmap 2024-03-29T11:25:30.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.498+0000 7f876134b640 10 monclient: monmap: 2024-03-29T11:25:30.484 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:30.484 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:30.484 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:25:30.500471+0000 2024-03-29T11:25:30.484 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:25:30.500471+0000 2024-03-29T11:25:30.484 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:30.484 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:30.484 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:30.484 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:25:30.484 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:25:30.484 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:30.487 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.502+0000 7f876134b640 1 Processor -- start 2024-03-29T11:25:30.487 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.502+0000 7f876134b640 1 -- start start 2024-03-29T11:25:30.487 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.502+0000 7f876134b640 10 monclient: init 2024-03-29T11:25:30.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.503+0000 7f876134b640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:30.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.503+0000 7f876134b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:30.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.503+0000 7f876134b640 10 monclient(hunting): picked mon.noname-b con 0x7f875c071820 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:30.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.503+0000 7f876134b640 10 monclient(hunting): picked mon.noname-c con 0x7f875c0af660 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:30.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.503+0000 7f876134b640 10 monclient(hunting): picked mon.noname-a con 0x7f875c06ab70 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:30.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.503+0000 7f876134b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875c0729c0 con 0x7f875c06ab70 2024-03-29T11:25:30.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.503+0000 7f876134b640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875c0afaf0 con 0x7f875c0af660 2024-03-29T11:25:30.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.503+0000 7f876134b640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875c0b1110 con 0x7f875c071820 2024-03-29T11:25:30.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.503+0000 7f876134b640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:30.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.503+0000 7f876134b640 10 monclient(hunting): authenticate will time out at 1190.800674s 2024-03-29T11:25:30.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.506+0000 7f875affd640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f875c071820 0x7f875c142090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:52966/0 (socket says 172.21.15.67:52966) 2024-03-29T11:25:30.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.506+0000 7f875affd640 1 -- 172.21.15.67:0/1281244603 learned_addr learned my addr 172.21.15.67:0/1281244603 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:30.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.507+0000 7f8759ffb640 1 -- 172.21.15.67:0/1281244603 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012863177 0 0) 0x7f875c0b1110 con 0x7f875c071820 2024-03-29T11:25:30.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.507+0000 7f8759ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:30.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.507+0000 7f8759ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:30.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.507+0000 7f8759ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:30.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.507+0000 7f8759ffb640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:25:30.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.507+0000 7f8759ffb640 1 -- 172.21.15.67:0/1281244603 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8744003520 con 0x7f875c071820 2024-03-29T11:25:30.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.508+0000 7f8759ffb640 1 -- 172.21.15.67:0/1281244603 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274917754 0 0) 0x7f875c0afaf0 con 0x7f875c0af660 2024-03-29T11:25:30.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.508+0000 7f8759ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:30.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.508+0000 7f8759ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:30.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.508+0000 7f8759ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:30.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.508+0000 7f8759ffb640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:25:30.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.508+0000 7f8759ffb640 1 -- 172.21.15.67:0/1281244603 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f875c0b1110 con 0x7f875c0af660 2024-03-29T11:25:30.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 1 -- 172.21.15.67:0/1281244603 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3387024446 0 0) 0x7f875c0b1110 con 0x7f875c0af660 2024-03-29T11:25:30.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 1 -- 172.21.15.67:0/1281244603 >> v1:172.21.15.67:6790/0 conn(0x7f875c071820 legacy=0x7f875c142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:30.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 1 -- 172.21.15.67:0/1281244603 >> v1:172.21.15.67:6789/0 conn(0x7f875c06ab70 legacy=0x7f875c14d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:30.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:30.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 1 monclient: found mon.noname-c 2024-03-29T11:25:30.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:30.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:25:30.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 1 -- 172.21.15.67:0/1281244603 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c0b12a0 con 0x7f875c0af660 2024-03-29T11:25:30.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:30.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:30.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:30.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f876134b640 5 monclient: authenticate success, global_id 4399 2024-03-29T11:25:30.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f876134b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:30.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 1 -- 172.21.15.67:0/1281244603 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f8750001670 con 0x7f875c0af660 2024-03-29T11:25:30.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:30.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:30.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f8759ffb640 10 monclient: dump: 2024-03-29T11:25:30.497 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:30.497 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:30.497 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:30.497 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:30.497 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:30.497 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:30.497 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:30.497 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:30.498 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:30.498 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:30.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.511+0000 7f876134b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:30.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.513+0000 7f8759ffb640 1 -- 172.21.15.67:0/1281244603 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8750001890 con 0x7f875c0af660 2024-03-29T11:25:30.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.513+0000 7f8759ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:30.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.513+0000 7f876134b640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:30.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.513+0000 7f876134b640 4 set_mon_vals no callback set 2024-03-29T11:25:30.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.513+0000 7f876134b640 10 monclient: shutdown 2024-03-29T11:25:30.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.513+0000 7f876134b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.513+0000 7f876134b640 1 -- 172.21.15.67:0/1281244603 >> v1:172.21.15.82:6789/0 conn(0x7f875c0af660 legacy=0x7f875c072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.514+0000 7f876134b640 1 -- 172.21.15.67:0/1281244603 shutdown_connections 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.514+0000 7f876134b640 1 -- 172.21.15.67:0/1281244603 >> 172.21.15.67:0/1281244603 conn(0x7f875c0b4930 msgr2=0x7f875c0b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.514+0000 7f876134b640 1 -- 172.21.15.67:0/1281244603 shutdown_connections 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.514+0000 7f876134b640 1 -- 172.21.15.67:0/1281244603 wait complete. 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.515+0000 7f876134b640 10 monclient: build_initial_monmap 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.515+0000 7f876134b640 10 monclient: monmap: 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:30.501 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:25:30.502 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:25:30.502 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:30.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.515+0000 7f876134b640 1 librados: starting msgr at 2024-03-29T11:25:30.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.515+0000 7f876134b640 1 librados: starting objecter 2024-03-29T11:25:30.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.516+0000 7f876134b640 1 Processor -- start 2024-03-29T11:25:30.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.516+0000 7f876134b640 1 -- start start 2024-03-29T11:25:30.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.516+0000 7f876134b640 1 librados: setting wanted keys 2024-03-29T11:25:30.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.516+0000 7f876134b640 1 librados: calling monclient init 2024-03-29T11:25:30.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.516+0000 7f876134b640 10 monclient: init 2024-03-29T11:25:30.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.516+0000 7f876134b640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:30.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.516+0000 7f876134b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:30.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.516+0000 7f876134b640 10 monclient(hunting): picked mon.noname-c con 0x7f875c06ab70 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:30.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.517+0000 7f876134b640 10 monclient(hunting): picked mon.noname-a con 0x7f875c071820 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:30.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.517+0000 7f876134b640 10 monclient(hunting): picked mon.noname-b con 0x7f875c0af660 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:30.505 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.517+0000 7f876134b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875c14b420 con 0x7f875c071820 2024-03-29T11:25:30.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.517+0000 7f876134b640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875c14b700 con 0x7f875c0af660 2024-03-29T11:25:30.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.517+0000 7f876134b640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875c149460 con 0x7f875c06ab70 2024-03-29T11:25:30.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.517+0000 7f876134b640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:30.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.517+0000 7f876134b640 10 monclient(hunting): authenticate will time out at 1190.813965s 2024-03-29T11:25:30.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.520+0000 7f875affd640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f875c06ab70 0x7f875c14de70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:54614/0 (socket says 172.21.15.67:54614) 2024-03-29T11:25:30.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.520+0000 7f875affd640 1 -- 172.21.15.67:0/606526320 learned_addr learned my addr 172.21.15.67:0/606526320 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:30.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.521+0000 7f87437fe640 10 client.?.objecter ms_handle_connect 0x7f875c06ab70 2024-03-29T11:25:30.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.521+0000 7f87437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:30.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.521+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547896892 0 0) 0x7f875c149460 con 0x7f875c06ab70 2024-03-29T11:25:30.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.521+0000 7f87437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:30.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.521+0000 7f87437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:30.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.521+0000 7f87437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:30.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.521+0000 7f87437fe640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:25:30.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.521+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8748004eb0 con 0x7f875c06ab70 2024-03-29T11:25:30.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.523+0000 7f87437fe640 10 client.?.objecter ms_handle_connect 0x7f875c0af660 2024-03-29T11:25:30.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.523+0000 7f87437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:30.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.524+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 302921016 0 0) 0x7f8748004eb0 con 0x7f875c06ab70 2024-03-29T11:25:30.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.524+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f875c149460 con 0x7f875c06ab70 2024-03-29T11:25:30.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.524+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f874c001950 con 0x7f875c06ab70 2024-03-29T11:25:30.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.524+0000 7f87437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:30.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.524+0000 7f87437fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:30.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.524+0000 7f87437fe640 10 monclient(hunting): dump: 2024-03-29T11:25:30.509 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:30.509 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:30.509 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:30.509 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:30.509 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:30.509 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:30.510 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:30.510 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:30.510 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:30.510 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:30.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.524+0000 7f87437fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:30.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.524+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1308086343 0 0) 0x7f875c149460 con 0x7f875c06ab70 2024-03-29T11:25:30.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.524+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 >> v1:172.21.15.82:6789/0 conn(0x7f875c0af660 legacy=0x7f875c14a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:30.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.524+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 >> v1:172.21.15.67:6789/0 conn(0x7f875c071820 legacy=0x7f875c149980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:30.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f87437fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:30.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f87437fe640 1 monclient: found mon.c 2024-03-29T11:25:30.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f87437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:30.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f87437fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:30.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c1490a0 con 0x7f875c06ab70 2024-03-29T11:25:30.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f87437fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:30.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f87437fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:30.512 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f87437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:30.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f876134b640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:25:30.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f876134b640 10 monclient: _renew_subs 2024-03-29T11:25:30.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f876134b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:30.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f876134b640 1 -- 172.21.15.67:0/606526320 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f875c1db130 con 0x7f875c06ab70 2024-03-29T11:25:30.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f876134b640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:30.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f876134b640 10 monclient: _renew_subs 2024-03-29T11:25:30.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f876134b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:30.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f876134b640 1 -- 172.21.15.67:0/606526320 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f875c1da8d0 con 0x7f875c06ab70 2024-03-29T11:25:30.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.526+0000 7f876134b640 1 librados: init done 2024-03-29T11:25:30.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.527+0000 7f876134b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:30.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.528+0000 7f876134b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:30.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.528+0000 7f876134b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:30.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.528+0000 7f876134b640 1 -- 172.21.15.67:0/606526320 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f875c1495f0 con 0x7f875c06ab70 2024-03-29T11:25:30.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.528+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f874c0047e0 con 0x7f875c06ab70 2024-03-29T11:25:30.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.528+0000 7f87437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:30.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.528+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f874c004bf0 con 0x7f875c06ab70 2024-03-29T11:25:30.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.528+0000 7f87437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:30.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.528+0000 7f87437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:30.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.528+0000 7f87437fe640 10 monclient: dump: 2024-03-29T11:25:30.516 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:30.516 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:30.516 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:30.516 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:30.516 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:30.516 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:30.516 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:30.516 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:30.516 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:30.516 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:30.516 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.528+0000 7f87597fa640 4 set_mon_vals no callback set 2024-03-29T11:25:30.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.529+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f874c004e10 con 0x7f875c06ab70 2024-03-29T11:25:30.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.530+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f874c084b40 con 0x7f875c06ab70 2024-03-29T11:25:30.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.530+0000 7f87437fe640 10 client.4478.objecter ms_dispatch 0x7f875c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:30.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.530+0000 7f87437fe640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:30.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.530+0000 7f87437fe640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:30.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.530+0000 7f87437fe640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-29T11:25:30.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.530+0000 7f875a7fc640 10 monclient: get_auth_request con 0x7f874806dcc0 auth_method 0 2024-03-29T11:25:30.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.530+0000 7f87437fe640 10 client.4478.objecter ms_handle_connect 0x7f874806dcc0 2024-03-29T11:25:30.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.538+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f874c050420 con 0x7f875c06ab70 2024-03-29T11:25:30.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.538+0000 7f87437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:30.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.538+0000 7f87437fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:30.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.862+0000 7f876134b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:30.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.862+0000 7f876134b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:30.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.862+0000 7f876134b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:30.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.863+0000 7f876134b640 1 -- 172.21.15.67:0/606526320 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f875c075f10 con 0x7f875c06ab70 2024-03-29T11:25:30.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.864+0000 7f87437fe640 1 -- 172.21.15.67:0/606526320 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f874c052350 con 0x7f875c06ab70 2024-03-29T11:25:30.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.864+0000 7f87437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:30.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.864+0000 7f87437fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:30.852 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:25:30.853 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:30.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.868+0000 7f876134b640 10 librados: watch_flush enter 2024-03-29T11:25:30.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.868+0000 7f876134b640 10 librados: watch_flush exit 2024-03-29T11:25:30.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.868+0000 7f876134b640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-29T11:25:30.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.868+0000 7f876134b640 10 client.4478.objecter successfully canceled tick 2024-03-29T11:25:30.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.868+0000 7f876134b640 1 -- 172.21.15.67:0/606526320 >> v1:172.21.15.67:6816/404787197 conn(0x7f874806dcc0 legacy=0x7f8748070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:30.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.868+0000 7f876134b640 10 monclient: shutdown 2024-03-29T11:25:30.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.868+0000 7f876134b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:30.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.868+0000 7f876134b640 1 -- 172.21.15.67:0/606526320 >> v1:172.21.15.67:6790/0 conn(0x7f875c06ab70 legacy=0x7f875c14de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:30.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.868+0000 7f876134b640 1 -- 172.21.15.67:0/606526320 shutdown_connections 2024-03-29T11:25:30.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.868+0000 7f876134b640 1 -- 172.21.15.67:0/606526320 >> 172.21.15.67:0/606526320 conn(0x7f875c0b4930 msgr2=0x7f875c071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:30.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.869+0000 7f876134b640 1 -- 172.21.15.67:0/606526320 shutdown_connections 2024-03-29T11:25:30.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.869+0000 7f876134b640 1 -- 172.21.15.67:0/606526320 wait complete. 2024-03-29T11:25:30.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:30.869+0000 7f876134b640 1 librados: shutdown 2024-03-29T11:25:30.872 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:37.679 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:37.824 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.838+0000 7ff8e71c6640 10 monclient: get_monmap_and_config 2024-03-29T11:25:37.824 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.838+0000 7ff8e71c6640 10 monclient: build_initial_monmap 2024-03-29T11:25:37.824 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.838+0000 7ff8e71c6640 10 monclient: monmap: 2024-03-29T11:25:37.824 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:37.824 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:37.824 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:25:37.840516+0000 2024-03-29T11:25:37.824 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:25:37.840516+0000 2024-03-29T11:25:37.824 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:37.824 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:37.825 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:37.825 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:25:37.825 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:25:37.825 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:37.827 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.842+0000 7ff8e71c6640 1 Processor -- start 2024-03-29T11:25:37.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.842+0000 7ff8e71c6640 1 -- start start 2024-03-29T11:25:37.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.842+0000 7ff8e71c6640 10 monclient: init 2024-03-29T11:25:37.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.843+0000 7ff8e71c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:37.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.843+0000 7ff8e71c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:37.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.843+0000 7ff8e71c6640 10 monclient(hunting): picked mon.noname-c con 0x7ff8e00b4790 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:37.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.843+0000 7ff8e71c6640 10 monclient(hunting): picked mon.noname-a con 0x7ff8e00aea90 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:37.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.843+0000 7ff8e71c6640 10 monclient(hunting): picked mon.noname-b con 0x7ff8e00b39f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:37.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.843+0000 7ff8e71c6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e00b0c60 con 0x7ff8e00aea90 2024-03-29T11:25:37.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.843+0000 7ff8e71c6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e00af020 con 0x7ff8e00b4790 2024-03-29T11:25:37.832 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.843+0000 7ff8e71c6640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e013fbd0 con 0x7ff8e00b39f0 2024-03-29T11:25:37.832 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.843+0000 7ff8e71c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:37.832 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.843+0000 7ff8e71c6640 10 monclient(hunting): authenticate will time out at 1198.140722s 2024-03-29T11:25:37.833 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.846+0000 7ff8e5763640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7ff8e00b39f0 0x7ff8e013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:41992/0 (socket says 172.21.15.67:41992) 2024-03-29T11:25:37.833 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.846+0000 7ff8e5763640 1 -- 172.21.15.67:0/3516073220 learned_addr learned my addr 172.21.15.67:0/3516073220 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:37.833 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.846+0000 7ff8df7fe640 1 -- 172.21.15.67:0/3516073220 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111464871 0 0) 0x7ff8e013fbd0 con 0x7ff8e00b39f0 2024-03-29T11:25:37.833 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:37.833 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:37.833 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:37.833 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:25:37.833 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 1 -- 172.21.15.67:0/3516073220 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8c8003520 con 0x7ff8e00b39f0 2024-03-29T11:25:37.834 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 1 -- 172.21.15.67:0/3516073220 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160920027 0 0) 0x7ff8e00b0c60 con 0x7ff8e00aea90 2024-03-29T11:25:37.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:37.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:37.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:37.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:25:37.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 1 -- 172.21.15.67:0/3516073220 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8e013fbd0 con 0x7ff8e00aea90 2024-03-29T11:25:37.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 1 -- 172.21.15.67:0/3516073220 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 523229954 0 0) 0x7ff8c8003520 con 0x7ff8e00b39f0 2024-03-29T11:25:37.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 1 -- 172.21.15.67:0/3516073220 >> v1:172.21.15.82:6789/0 conn(0x7ff8e00b4790 legacy=0x7ff8e00b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:37.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 1 -- 172.21.15.67:0/3516073220 >> v1:172.21.15.67:6789/0 conn(0x7ff8e00aea90 legacy=0x7ff8e00af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:37.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 1 monclient: found mon.noname-b 2024-03-29T11:25:37.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:37.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:25:37.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 1 -- 172.21.15.67:0/3516073220 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e013fd60 con 0x7ff8e00b39f0 2024-03-29T11:25:37.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:37.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:37.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:37.840 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 1 -- 172.21.15.67:0/3516073220 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff8d8001a10 con 0x7ff8e00b39f0 2024-03-29T11:25:37.840 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:37.840 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:37.840 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient: dump: 2024-03-29T11:25:37.840 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:37.840 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:37.840 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:37.840 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:37.840 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:37.840 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:37.840 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:37.840 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:37.840 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:37.840 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8e71c6640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8e71c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 1 -- 172.21.15.67:0/3516073220 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8d8001c60 con 0x7ff8e00b39f0 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 1 -- 172.21.15.67:0/3516073220 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff8d8001ff0 con 0x7ff8e00b39f0 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8df7fe640 10 monclient: dump: 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:37.841 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:37.842 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:37.842 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:37.842 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:37.842 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8e71c6640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:37.842 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.847+0000 7ff8e71c6640 4 set_mon_vals no callback set 2024-03-29T11:25:37.842 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.848+0000 7ff8e71c6640 10 monclient: shutdown 2024-03-29T11:25:37.842 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.848+0000 7ff8e71c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:37.842 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.848+0000 7ff8e71c6640 1 -- 172.21.15.67:0/3516073220 >> v1:172.21.15.67:6790/0 conn(0x7ff8e00b39f0 legacy=0x7ff8e013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.842 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.848+0000 7ff8e71c6640 1 -- 172.21.15.67:0/3516073220 shutdown_connections 2024-03-29T11:25:37.842 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.848+0000 7ff8e71c6640 1 -- 172.21.15.67:0/3516073220 >> 172.21.15.67:0/3516073220 conn(0x7ff8e0066a00 msgr2=0x7ff8e0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:37.842 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.849+0000 7ff8e71c6640 1 -- 172.21.15.67:0/3516073220 shutdown_connections 2024-03-29T11:25:37.842 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.849+0000 7ff8e71c6640 1 -- 172.21.15.67:0/3516073220 wait complete. 2024-03-29T11:25:37.842 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.851+0000 7ff8e71c6640 10 monclient: build_initial_monmap 2024-03-29T11:25:37.843 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.851+0000 7ff8e71c6640 10 monclient: monmap: 2024-03-29T11:25:37.843 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:37.843 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:37.843 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:25:37.843 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:25:37.843 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:37.843 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:37.843 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:37.843 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:25:37.843 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:25:37.843 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:37.843 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.851+0000 7ff8e71c6640 1 librados: starting msgr at 2024-03-29T11:25:37.843 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.851+0000 7ff8e71c6640 1 librados: starting objecter 2024-03-29T11:25:37.843 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.852+0000 7ff8e71c6640 1 Processor -- start 2024-03-29T11:25:37.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.852+0000 7ff8e71c6640 1 -- start start 2024-03-29T11:25:37.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.852+0000 7ff8e71c6640 1 librados: setting wanted keys 2024-03-29T11:25:37.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.852+0000 7ff8e71c6640 1 librados: calling monclient init 2024-03-29T11:25:37.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.852+0000 7ff8e71c6640 10 monclient: init 2024-03-29T11:25:37.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.853+0000 7ff8e71c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:37.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.853+0000 7ff8e71c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:37.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.854+0000 7ff8e71c6640 10 monclient(hunting): picked mon.noname-a con 0x7ff8e00b39f0 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:37.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.854+0000 7ff8e71c6640 10 monclient(hunting): picked mon.noname-c con 0x7ff8e00b4790 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:37.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.854+0000 7ff8e71c6640 10 monclient(hunting): picked mon.noname-b con 0x7ff8e006f180 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:37.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.854+0000 7ff8e71c6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e00aa540 con 0x7ff8e00b39f0 2024-03-29T11:25:37.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.854+0000 7ff8e71c6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e00a99a0 con 0x7ff8e006f180 2024-03-29T11:25:37.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.854+0000 7ff8e71c6640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e0071310 con 0x7ff8e00b4790 2024-03-29T11:25:37.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.854+0000 7ff8e71c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:37.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.854+0000 7ff8e71c6640 10 monclient(hunting): authenticate will time out at 1198.151576s 2024-03-29T11:25:37.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.854+0000 7ff8dffff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7ff8e00b4790 0x7ff8e00a9e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:43360/0 (socket says 172.21.15.67:43360) 2024-03-29T11:25:37.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.854+0000 7ff8dffff640 1 -- 172.21.15.67:0/1987169946 learned_addr learned my addr 172.21.15.67:0/1987169946 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:37.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 10 client.?.objecter ms_handle_connect 0x7ff8e00b4790 2024-03-29T11:25:37.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:37.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287109208 0 0) 0x7ff8e0071310 con 0x7ff8e00b4790 2024-03-29T11:25:37.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:37.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:37.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:37.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:25:37.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8cc004eb0 con 0x7ff8e00b4790 2024-03-29T11:25:37.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2835010261 0 0) 0x7ff8cc004eb0 con 0x7ff8e00b4790 2024-03-29T11:25:37.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8e0071310 con 0x7ff8e00b4790 2024-03-29T11:25:37.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 10 client.?.objecter ms_handle_connect 0x7ff8e006f180 2024-03-29T11:25:37.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:37.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff8d4001a10 con 0x7ff8e00b4790 2024-03-29T11:25:37.850 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:37.850 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:37.850 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 10 monclient(hunting): dump: 2024-03-29T11:25:37.850 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:37.850 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:37.850 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:37.850 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:37.850 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:37.850 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:37.850 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:37.850 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:37.850 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:37.850 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:37.850 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.855+0000 7ff8dd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:37.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627063771 0 0) 0x7ff8e00a99a0 con 0x7ff8e006f180 2024-03-29T11:25:37.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:37.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:37.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:37.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:25:37.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8cc004eb0 con 0x7ff8e006f180 2024-03-29T11:25:37.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2546169479 0 0) 0x7ff8cc004eb0 con 0x7ff8e006f180 2024-03-29T11:25:37.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8e00a99a0 con 0x7ff8e006f180 2024-03-29T11:25:37.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff8d8001dd0 con 0x7ff8e006f180 2024-03-29T11:25:37.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:37.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:37.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 10 monclient(hunting): dump: 2024-03-29T11:25:37.851 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:37.851 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:37.852 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:37.852 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:37.852 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:37.852 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:37.852 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:37.852 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:37.852 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:37.852 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:37.852 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 62271627 0 0) 0x7ff8e0071310 con 0x7ff8e00b4790 2024-03-29T11:25:37.852 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 >> v1:172.21.15.82:6789/0 conn(0x7ff8e006f180 legacy=0x7ff8e01e6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.852 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.856+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 >> v1:172.21.15.67:6789/0 conn(0x7ff8e00b39f0 legacy=0x7ff8e0070a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:37.852 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.857+0000 7ff8dd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:37.852 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.857+0000 7ff8dd7fa640 1 monclient: found mon.c 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.857+0000 7ff8dd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.857+0000 7ff8dd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.857+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e0071600 con 0x7ff8e00b4790 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.857+0000 7ff8dd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.857+0000 7ff8dd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.857+0000 7ff8dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.858+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8d4001d10 con 0x7ff8e00b4790 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.858+0000 7ff8dd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.858+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff8d4003a60 con 0x7ff8e00b4790 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.858+0000 7ff8dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.858+0000 7ff8dd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.858+0000 7ff8dd7fa640 10 monclient: dump: 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:37.853 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:37.854 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:37.854 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:37.854 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:37.854 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:37.854 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:37.854 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:37.854 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:37.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.858+0000 7ff8deffd640 4 set_mon_vals no callback set 2024-03-29T11:25:37.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.859+0000 7ff8e71c6640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:25:37.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.859+0000 7ff8e71c6640 10 monclient: _renew_subs 2024-03-29T11:25:37.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.859+0000 7ff8e71c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:37.854 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.859+0000 7ff8e71c6640 1 -- 172.21.15.67:0/1987169946 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e0070100 con 0x7ff8e00b4790 2024-03-29T11:25:37.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.859+0000 7ff8e71c6640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:37.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.859+0000 7ff8e71c6640 10 monclient: _renew_subs 2024-03-29T11:25:37.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.859+0000 7ff8e71c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:37.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.859+0000 7ff8e71c6640 1 -- 172.21.15.67:0/1987169946 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e01e7d70 con 0x7ff8e00b4790 2024-03-29T11:25:37.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.859+0000 7ff8e71c6640 1 librados: init done 2024-03-29T11:25:37.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.860+0000 7ff8e71c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:37.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.860+0000 7ff8e71c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:37.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.860+0000 7ff8e71c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:37.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.860+0000 7ff8e71c6640 1 -- 172.21.15.67:0/1987169946 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e006eea0 con 0x7ff8e00b4790 2024-03-29T11:25:37.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.862+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7ff8d4003bc0 con 0x7ff8e00b4790 2024-03-29T11:25:37.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.862+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7ff8d4084b40 con 0x7ff8e00b4790 2024-03-29T11:25:37.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.862+0000 7ff8dd7fa640 10 client.4484.objecter ms_dispatch 0x7ff8e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:37.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.862+0000 7ff8dd7fa640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:37.855 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.862+0000 7ff8dd7fa640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:37.856 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.862+0000 7ff8dd7fa640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-29T11:25:37.856 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.862+0000 7ff8e4f62640 10 monclient: get_auth_request con 0x7ff8cc06de60 auth_method 0 2024-03-29T11:25:37.856 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.863+0000 7ff8dd7fa640 10 client.4484.objecter ms_handle_connect 0x7ff8cc06de60 2024-03-29T11:25:37.856 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.870+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8e006eea0 con 0x7ff8e00b4790 2024-03-29T11:25:37.856 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.870+0000 7ff8dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:37.856 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:37.870+0000 7ff8dd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:38.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.193+0000 7ff8e71c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.179 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.193+0000 7ff8e71c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.179 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.193+0000 7ff8e71c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:25:38.179 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.193+0000 7ff8e71c6640 1 -- 172.21.15.67:0/1987169946 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8e006eea0 con 0x7ff8e00b4790 2024-03-29T11:25:38.181 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.196+0000 7ff8dd7fa640 1 -- 172.21.15.67:0/1987169946 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7ff8e006eea0 con 0x7ff8e00b4790 2024-03-29T11:25:38.181 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.196+0000 7ff8dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.181 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.196+0000 7ff8dd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:38.182 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:25:38.182 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:38.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.201+0000 7ff8e71c6640 10 librados: watch_flush enter 2024-03-29T11:25:38.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.201+0000 7ff8e71c6640 10 librados: watch_flush exit 2024-03-29T11:25:38.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.201+0000 7ff8e71c6640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-29T11:25:38.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.201+0000 7ff8e71c6640 10 client.4484.objecter successfully canceled tick 2024-03-29T11:25:38.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.202+0000 7ff8e71c6640 1 -- 172.21.15.67:0/1987169946 >> v1:172.21.15.67:6816/404787197 conn(0x7ff8cc06de60 legacy=0x7ff8cc070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.202+0000 7ff8e71c6640 10 monclient: shutdown 2024-03-29T11:25:38.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.202+0000 7ff8e71c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:38.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.202+0000 7ff8e71c6640 1 -- 172.21.15.67:0/1987169946 >> v1:172.21.15.67:6790/0 conn(0x7ff8e00b4790 legacy=0x7ff8e00a9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.202+0000 7ff8e71c6640 1 -- 172.21.15.67:0/1987169946 shutdown_connections 2024-03-29T11:25:38.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.202+0000 7ff8e71c6640 1 -- 172.21.15.67:0/1987169946 >> 172.21.15.67:0/1987169946 conn(0x7ff8e0066a00 msgr2=0x7ff8e00b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:38.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.202+0000 7ff8e71c6640 1 -- 172.21.15.67:0/1987169946 shutdown_connections 2024-03-29T11:25:38.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.202+0000 7ff8e71c6640 1 -- 172.21.15.67:0/1987169946 wait complete. 2024-03-29T11:25:38.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:38.203+0000 7ff8e71c6640 1 librados: shutdown 2024-03-29T11:25:38.206 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:45.013 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:45.164 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.179+0000 7fe2f3b52640 10 monclient: get_monmap_and_config 2024-03-29T11:25:45.164 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.179+0000 7fe2f3b52640 10 monclient: build_initial_monmap 2024-03-29T11:25:45.164 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.179+0000 7fe2f3b52640 10 monclient: monmap: 2024-03-29T11:25:45.164 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:45.165 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:45.165 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:25:45.180066+0000 2024-03-29T11:25:45.165 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:25:45.180066+0000 2024-03-29T11:25:45.165 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:45.165 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:45.165 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:45.165 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:25:45.165 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:25:45.165 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:45.165 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.180+0000 7fe2f3b52640 1 Processor -- start 2024-03-29T11:25:45.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.181+0000 7fe2f3b52640 1 -- start start 2024-03-29T11:25:45.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.181+0000 7fe2f3b52640 10 monclient: init 2024-03-29T11:25:45.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.182+0000 7fe2f3b52640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:45.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.182+0000 7fe2f3b52640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:45.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.182+0000 7fe2f3b52640 10 monclient(hunting): picked mon.noname-b con 0x7fe2ec0a9ac0 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:45.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.182+0000 7fe2f3b52640 10 monclient(hunting): picked mon.noname-a con 0x7fe2ec13ee90 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:45.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.182+0000 7fe2f3b52640 10 monclient(hunting): picked mon.noname-c con 0x7fe2ec0af900 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:45.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.182+0000 7fe2f3b52640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2ec0afd10 con 0x7fe2ec13ee90 2024-03-29T11:25:45.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.182+0000 7fe2f3b52640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2ec0af6f0 con 0x7fe2ec0af900 2024-03-29T11:25:45.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.182+0000 7fe2f3b52640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2ec0aac80 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.182+0000 7fe2f3b52640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:45.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.182+0000 7fe2f3b52640 10 monclient(hunting): authenticate will time out at 1205.478149s 2024-03-29T11:25:45.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.186+0000 7fe2f20ef640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fe2ec0af900 0x7fe2ec149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:35503/0 (socket says 172.21.15.67:35503) 2024-03-29T11:25:45.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.186+0000 7fe2f20ef640 1 -- 172.21.15.67:0/1928042763 learned_addr learned my addr 172.21.15.67:0/1928042763 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:45.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.187+0000 7fe2f08ec640 1 -- 172.21.15.67:0/1928042763 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774426832 0 0) 0x7fe2ec0af6f0 con 0x7fe2ec0af900 2024-03-29T11:25:45.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.187+0000 7fe2f08ec640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.187+0000 7fe2f08ec640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.187+0000 7fe2f08ec640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.187+0000 7fe2f08ec640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:25:45.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.187+0000 7fe2f08ec640 1 -- 172.21.15.67:0/1928042763 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d8003520 con 0x7fe2ec0af900 2024-03-29T11:25:45.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.187+0000 7fe2f08ec640 1 -- 172.21.15.67:0/1928042763 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732806395 0 0) 0x7fe2ec0aac80 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.187+0000 7fe2f08ec640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.187+0000 7fe2f08ec640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.187+0000 7fe2f08ec640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.187+0000 7fe2f08ec640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.187+0000 7fe2f08ec640 1 -- 172.21.15.67:0/1928042763 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2ec0af6f0 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 1 -- 172.21.15.67:0/1928042763 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 621855839 0 0) 0x7fe2d8003520 con 0x7fe2ec0af900 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 1 -- 172.21.15.67:0/1928042763 >> v1:172.21.15.67:6790/0 conn(0x7fe2ec0a9ac0 legacy=0x7fe2ec0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 1 -- 172.21.15.67:0/1928042763 >> v1:172.21.15.67:6789/0 conn(0x7fe2ec13ee90 legacy=0x7fe2ec13b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 10 monclient: _finish_hunting 0 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 1 monclient: found mon.noname-c 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 1 -- 172.21.15.67:0/1928042763 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ec0aaf60 con 0x7fe2ec0af900 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 10 monclient: _finish_auth 0 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 10 monclient: _check_auth_tickets 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:45.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 1 -- 172.21.15.67:0/1928042763 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe2e4005030 con 0x7fe2ec0af900 2024-03-29T11:25:45.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f3b52640 5 monclient: authenticate success, global_id 4408 2024-03-29T11:25:45.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f3b52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:45.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:45.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 10 monclient: dump: 2024-03-29T11:25:45.178 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:45.178 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:45.178 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:45.178 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:45.178 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.179 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:45.179 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:45.179 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:45.179 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:45.179 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:45.179 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f3b52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:45.179 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 1 -- 172.21.15.67:0/1928042763 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2e4005250 con 0x7fe2ec0af900 2024-03-29T11:25:45.179 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:45.179 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 1 -- 172.21.15.67:0/1928042763 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe2e4001770 con 0x7fe2ec0af900 2024-03-29T11:25:45.179 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.179 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:45.179 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f08ec640 10 monclient: dump: 2024-03-29T11:25:45.179 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:45.179 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f3b52640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.188+0000 7fe2f3b52640 4 set_mon_vals no callback set 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.193+0000 7fe2f3b52640 10 monclient: shutdown 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.193+0000 7fe2f3b52640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.193+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1928042763 >> v1:172.21.15.82:6789/0 conn(0x7fe2ec0af900 legacy=0x7fe2ec149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.193+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1928042763 shutdown_connections 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.193+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1928042763 >> 172.21.15.67:0/1928042763 conn(0x7fe2ec0b35e0 msgr2=0x7fe2ec0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:45.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.193+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1928042763 shutdown_connections 2024-03-29T11:25:45.181 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.193+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1928042763 wait complete. 2024-03-29T11:25:45.183 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.194+0000 7fe2f3b52640 10 monclient: build_initial_monmap 2024-03-29T11:25:45.183 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.194+0000 7fe2f3b52640 10 monclient: monmap: 2024-03-29T11:25:45.183 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:45.183 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:45.183 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:25:45.183 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:25:45.183 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:45.184 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:45.184 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:45.184 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:25:45.184 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:25:45.184 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:45.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.195+0000 7fe2f3b52640 1 librados: starting msgr at 2024-03-29T11:25:45.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.195+0000 7fe2f3b52640 1 librados: starting objecter 2024-03-29T11:25:45.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.195+0000 7fe2f3b52640 1 Processor -- start 2024-03-29T11:25:45.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.195+0000 7fe2f3b52640 1 -- start start 2024-03-29T11:25:45.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.195+0000 7fe2f3b52640 1 librados: setting wanted keys 2024-03-29T11:25:45.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.195+0000 7fe2f3b52640 1 librados: calling monclient init 2024-03-29T11:25:45.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.195+0000 7fe2f3b52640 10 monclient: init 2024-03-29T11:25:45.185 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.196+0000 7fe2f3b52640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:45.185 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.196+0000 7fe2f3b52640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:45.185 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.196+0000 7fe2f3b52640 10 monclient(hunting): picked mon.noname-b con 0x7fe2ec0a9ac0 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:45.185 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.201+0000 7fe2f3b52640 10 monclient(hunting): picked mon.noname-a con 0x7fe2ec0af900 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:45.185 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.201+0000 7fe2f3b52640 10 monclient(hunting): picked mon.noname-c con 0x7fe2ec13ee90 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:45.185 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.201+0000 7fe2f3b52640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2ec138aa0 con 0x7fe2ec0af900 2024-03-29T11:25:45.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.201+0000 7fe2f3b52640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2ec0713a0 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.201+0000 7fe2f3b52640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2ec070060 con 0x7fe2ec13ee90 2024-03-29T11:25:45.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.201+0000 7fe2f3b52640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:45.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.201+0000 7fe2f3b52640 10 monclient(hunting): authenticate will time out at 1205.497051s 2024-03-29T11:25:45.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.201+0000 7fe2f18ee640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fe2ec0a9ac0 0x7fe2ec1414f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:35579/0 (socket says 172.21.15.67:35579) 2024-03-29T11:25:45.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.201+0000 7fe2f18ee640 1 -- 172.21.15.67:0/1735891048 learned_addr learned my addr 172.21.15.67:0/1735891048 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:45.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.203+0000 7fe2e27fc640 10 client.?.objecter ms_handle_connect 0x7fe2ec0a9ac0 2024-03-29T11:25:45.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.203+0000 7fe2e27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:45.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.204+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022011844 0 0) 0x7fe2ec0713a0 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.204+0000 7fe2e27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.204+0000 7fe2e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.204+0000 7fe2e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.204+0000 7fe2e27fc640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:25:45.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.204+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d0004eb0 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.207+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3033147039 0 0) 0x7fe2d0004eb0 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.207+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2ec0713a0 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.207+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe2dc001a50 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.207+0000 7fe2e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.207+0000 7fe2e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:45.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.207+0000 7fe2e27fc640 10 monclient(hunting): dump: 2024-03-29T11:25:45.192 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:45.192 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:45.192 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:45.192 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:45.192 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.192 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:45.192 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:45.193 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:45.193 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:45.193 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:45.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.207+0000 7fe2e27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:45.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.207+0000 7fe2e27fc640 10 client.?.objecter ms_handle_connect 0x7fe2ec0af900 2024-03-29T11:25:45.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.207+0000 7fe2e27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:45.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1466792195 0 0) 0x7fe2ec0713a0 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 >> v1:172.21.15.67:6790/0 conn(0x7fe2ec13ee90 legacy=0x7fe2ec1ff860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:45.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 >> v1:172.21.15.67:6789/0 conn(0x7fe2ec0af900 legacy=0x7fe2ec070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2e27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:45.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2e27fc640 1 monclient: found mon.b 2024-03-29T11:25:45.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:45.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2e27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:25:45.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ec0701f0 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2e27fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:45.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2e27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:45.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:45.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2f3b52640 5 monclient: authenticate success, global_id 4411 2024-03-29T11:25:45.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2f3b52640 10 monclient: _renew_subs 2024-03-29T11:25:45.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2f3b52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:25:45.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1735891048 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2ec1f8270 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2f3b52640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:45.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2f3b52640 10 monclient: _renew_subs 2024-03-29T11:25:45.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2f3b52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:25:45.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1735891048 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2ec200f80 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.208+0000 7fe2f3b52640 1 librados: init done 2024-03-29T11:25:45.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.212+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2dc001cb0 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.212+0000 7fe2e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:45.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.212+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fe2dc003a40 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.212+0000 7fe2e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.212+0000 7fe2e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:45.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.212+0000 7fe2e27fc640 10 monclient: dump: 2024-03-29T11:25:45.196 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:45.196 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:45.196 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:45.197 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:45.197 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.197 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:45.197 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:45.197 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:45.197 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:45.197 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:45.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.212+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fe2dc003c60 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.212+0000 7fe2e3fff640 4 set_mon_vals no callback set 2024-03-29T11:25:45.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.212+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fe2dc084c20 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.212+0000 7fe2e27fc640 10 client.4411.objecter ms_dispatch 0x7fe2ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:45.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.212+0000 7fe2e27fc640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:45.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.212+0000 7fe2e27fc640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:45.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.212+0000 7fe2e27fc640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-29T11:25:45.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.213+0000 7fe2f10ed640 10 monclient: get_auth_request con 0x7fe2d006dc40 auth_method 0 2024-03-29T11:25:45.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.213+0000 7fe2e27fc640 10 client.4411.objecter ms_handle_connect 0x7fe2d006dc40 2024-03-29T11:25:45.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.214+0000 7fe2f3b52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.214+0000 7fe2f3b52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.214+0000 7fe2f3b52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:25:45.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.214+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1735891048 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d4005080 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.208 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.224+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe2dc050580 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.208 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.224+0000 7fe2e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.208 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.224+0000 7fe2e27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:45.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.562+0000 7fe2f3b52640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:45.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.562+0000 7fe2f3b52640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:45.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.562+0000 7fe2f3b52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:25:45.548 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.562+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1735891048 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2d4005330 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.548 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.563+0000 7fe2e27fc640 1 -- 172.21.15.67:0/1735891048 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fe2dc0524b0 con 0x7fe2ec0a9ac0 2024-03-29T11:25:45.548 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.563+0000 7fe2e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:45.548 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.563+0000 7fe2e27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:45.549 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:25:45.549 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:45.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.568+0000 7fe2f3b52640 10 librados: watch_flush enter 2024-03-29T11:25:45.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.568+0000 7fe2f3b52640 10 librados: watch_flush exit 2024-03-29T11:25:45.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.568+0000 7fe2f3b52640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-29T11:25:45.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.568+0000 7fe2f3b52640 10 client.4411.objecter successfully canceled tick 2024-03-29T11:25:45.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.568+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1735891048 >> v1:172.21.15.67:6816/404787197 conn(0x7fe2d006dc40 legacy=0x7fe2d0070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.568+0000 7fe2f3b52640 10 monclient: shutdown 2024-03-29T11:25:45.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.568+0000 7fe2f3b52640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:45.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.568+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1735891048 >> v1:172.21.15.82:6789/0 conn(0x7fe2ec0a9ac0 legacy=0x7fe2ec1414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.568+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1735891048 shutdown_connections 2024-03-29T11:25:45.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.568+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1735891048 >> 172.21.15.67:0/1735891048 conn(0x7fe2ec0b35e0 msgr2=0x7fe2ec0b2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:45.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.569+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1735891048 shutdown_connections 2024-03-29T11:25:45.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.569+0000 7fe2f3b52640 1 -- 172.21.15.67:0/1735891048 wait complete. 2024-03-29T11:25:45.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:45.569+0000 7fe2f3b52640 1 librados: shutdown 2024-03-29T11:25:45.571 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:52.378 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:52.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.547+0000 7feeb679d640 10 monclient: get_monmap_and_config 2024-03-29T11:25:52.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.547+0000 7feeb679d640 10 monclient: build_initial_monmap 2024-03-29T11:25:52.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.547+0000 7feeb679d640 10 monclient: monmap: 2024-03-29T11:25:52.532 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:52.533 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:52.533 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:25:52.548208+0000 2024-03-29T11:25:52.533 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:25:52.548208+0000 2024-03-29T11:25:52.533 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:52.533 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:52.533 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:52.533 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:25:52.533 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:25:52.533 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:52.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.548+0000 7feeb679d640 1 Processor -- start 2024-03-29T11:25:52.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.548+0000 7feeb679d640 1 -- start start 2024-03-29T11:25:52.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.548+0000 7feeb679d640 10 monclient: init 2024-03-29T11:25:52.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.549+0000 7feeb679d640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:52.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.549+0000 7feeb679d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:52.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.549+0000 7feeb679d640 10 monclient(hunting): picked mon.noname-b con 0x7feeb01457f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:52.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.553+0000 7feeb679d640 10 monclient(hunting): picked mon.noname-a con 0x7feeb013f650 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:52.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.553+0000 7feeaffff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7feeb01457f0 0x7feeb013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:35413/0 (socket says 172.21.15.67:35413) 2024-03-29T11:25:52.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.554+0000 7feeaffff640 1 -- 172.21.15.67:0/3445806975 learned_addr learned my addr 172.21.15.67:0/3445806975 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:52.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.554+0000 7feeb679d640 10 monclient(hunting): picked mon.noname-c con 0x7feeb00a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:52.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.554+0000 7feeb679d640 1 -- 172.21.15.67:0/3445806975 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb00aaef0 con 0x7feeb013f650 2024-03-29T11:25:52.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.554+0000 7feeb679d640 1 -- 172.21.15.67:0/3445806975 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb0145bd0 con 0x7feeb00a9d70 2024-03-29T11:25:52.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.554+0000 7feeb679d640 1 -- 172.21.15.67:0/3445806975 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb0145eb0 con 0x7feeb01457f0 2024-03-29T11:25:52.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.554+0000 7feeb679d640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:52.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.554+0000 7feeb679d640 10 monclient(hunting): authenticate will time out at 1212.850652s 2024-03-29T11:25:52.545 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.557+0000 7feeaeffd640 1 -- 172.21.15.67:0/3445806975 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488770854 0 0) 0x7feeb0145eb0 con 0x7feeb01457f0 2024-03-29T11:25:52.545 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.557+0000 7feeaeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:52.545 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.557+0000 7feeaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:52.545 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.557+0000 7feeaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:52.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.557+0000 7feeaeffd640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:25:52.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.557+0000 7feeaeffd640 1 -- 172.21.15.67:0/3445806975 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee9c003520 con 0x7feeb01457f0 2024-03-29T11:25:52.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.557+0000 7feeaeffd640 1 -- 172.21.15.67:0/3445806975 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815709807 0 0) 0x7feeb00aaef0 con 0x7feeb013f650 2024-03-29T11:25:52.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.557+0000 7feeaeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:52.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.557+0000 7feeaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:52.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.557+0000 7feeaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:52.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.557+0000 7feeaeffd640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:25:52.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.557+0000 7feeaeffd640 1 -- 172.21.15.67:0/3445806975 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeb0145eb0 con 0x7feeb013f650 2024-03-29T11:25:52.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.558+0000 7feeaeffd640 1 -- 172.21.15.67:0/3445806975 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795795022 0 0) 0x7feeb0145bd0 con 0x7feeb00a9d70 2024-03-29T11:25:52.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.558+0000 7feeaeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:52.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.558+0000 7feeaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:52.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.558+0000 7feeaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:52.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.558+0000 7feeaeffd640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:25:52.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.558+0000 7feeaeffd640 1 -- 172.21.15.67:0/3445806975 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeb00aaef0 con 0x7feeb00a9d70 2024-03-29T11:25:52.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.558+0000 7feeaeffd640 1 -- 172.21.15.67:0/3445806975 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2948233132 0 0) 0x7fee9c003520 con 0x7feeb01457f0 2024-03-29T11:25:52.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.558+0000 7feeaeffd640 1 -- 172.21.15.67:0/3445806975 >> v1:172.21.15.82:6789/0 conn(0x7feeb00a9d70 legacy=0x7feeb014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:52.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.558+0000 7feeaeffd640 1 -- 172.21.15.67:0/3445806975 >> v1:172.21.15.67:6789/0 conn(0x7feeb013f650 legacy=0x7feeb00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:52.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.558+0000 7feeaeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:52.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.558+0000 7feeaeffd640 1 monclient: found mon.noname-b 2024-03-29T11:25:52.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.558+0000 7feeaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:52.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.558+0000 7feeaeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:25:52.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeaeffd640 1 -- 172.21.15.67:0/3445806975 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb0146340 con 0x7feeb01457f0 2024-03-29T11:25:52.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeaeffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:52.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeaeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:52.548 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:52.548 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeaeffd640 1 -- 172.21.15.67:0/3445806975 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7feea8001a10 con 0x7feeb01457f0 2024-03-29T11:25:52.548 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:52.548 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeaeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:52.548 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeaeffd640 10 monclient: dump: 2024-03-29T11:25:52.548 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:52.548 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:52.548 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:52.548 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:52.548 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:52.548 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:52.548 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:52.548 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:52.549 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:52.549 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:52.549 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeaeffd640 1 -- 172.21.15.67:0/3445806975 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feea8001c30 con 0x7feeb01457f0 2024-03-29T11:25:52.549 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeaeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:52.549 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeb679d640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:25:52.549 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeb679d640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:52.549 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeb679d640 4 set_mon_vals no callback set 2024-03-29T11:25:52.549 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeaeffd640 1 -- 172.21.15.67:0/3445806975 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7feea8001fc0 con 0x7feeb01457f0 2024-03-29T11:25:52.549 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:52.549 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.559+0000 7feeaeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:52.549 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.560+0000 7feeaeffd640 10 monclient: dump: 2024-03-29T11:25:52.549 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:52.549 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:52.550 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:52.550 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:52.550 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:52.550 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:52.550 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:52.550 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:52.550 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:52.550 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:52.550 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.560+0000 7feeb679d640 10 monclient: shutdown 2024-03-29T11:25:52.550 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.560+0000 7feeb679d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:52.550 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.561+0000 7feeb679d640 1 -- 172.21.15.67:0/3445806975 >> v1:172.21.15.67:6790/0 conn(0x7feeb01457f0 legacy=0x7feeb013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:52.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.562+0000 7feeb679d640 1 -- 172.21.15.67:0/3445806975 shutdown_connections 2024-03-29T11:25:52.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.562+0000 7feeb679d640 1 -- 172.21.15.67:0/3445806975 >> 172.21.15.67:0/3445806975 conn(0x7feeb0066a00 msgr2=0x7feeb0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:52.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.563+0000 7feeb679d640 1 -- 172.21.15.67:0/3445806975 shutdown_connections 2024-03-29T11:25:52.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.563+0000 7feeb679d640 1 -- 172.21.15.67:0/3445806975 wait complete. 2024-03-29T11:25:52.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.564+0000 7feeb679d640 10 monclient: build_initial_monmap 2024-03-29T11:25:52.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.564+0000 7feeb679d640 10 monclient: monmap: 2024-03-29T11:25:52.552 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:52.552 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:52.553 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:25:52.553 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:25:52.553 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:52.553 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:52.553 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:52.553 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:25:52.553 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:25:52.553 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:52.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.564+0000 7feeb679d640 1 librados: starting msgr at 2024-03-29T11:25:52.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.564+0000 7feeb679d640 1 librados: starting objecter 2024-03-29T11:25:52.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.564+0000 7feeb679d640 1 Processor -- start 2024-03-29T11:25:52.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.565+0000 7feeb679d640 1 -- start start 2024-03-29T11:25:52.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.565+0000 7feeb679d640 1 librados: setting wanted keys 2024-03-29T11:25:52.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.565+0000 7feeb679d640 1 librados: calling monclient init 2024-03-29T11:25:52.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.565+0000 7feeb679d640 10 monclient: init 2024-03-29T11:25:52.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.566+0000 7feeb679d640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:52.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.566+0000 7feeb679d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:52.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.566+0000 7feeb679d640 10 monclient(hunting): picked mon.noname-b con 0x7feeb00a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:52.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.568+0000 7feeb679d640 10 monclient(hunting): picked mon.noname-a con 0x7feeb013f650 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:52.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.569+0000 7feeaffff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7feeb00a9d70 0x7feeb01db8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:56033/0 (socket says 172.21.15.67:56033) 2024-03-29T11:25:52.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.570+0000 7feeb679d640 10 monclient(hunting): picked mon.noname-c con 0x7feeb01457f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:52.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.570+0000 7feeaf7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7feeb013f650 0x7feeb00719e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46775/0 (socket says 172.21.15.67:46775) 2024-03-29T11:25:52.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.570+0000 7feeaf7fe640 1 -- 172.21.15.67:0/499693617 learned_addr learned my addr 172.21.15.67:0/499693617 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:52.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.570+0000 7feeb679d640 1 -- 172.21.15.67:0/499693617 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb01dc1c0 con 0x7feeb013f650 2024-03-29T11:25:52.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.570+0000 7feeb679d640 1 -- 172.21.15.67:0/499693617 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb0070f90 con 0x7feeb00a9d70 2024-03-29T11:25:52.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.570+0000 7feeb679d640 1 -- 172.21.15.67:0/499693617 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb0071270 con 0x7feeb01457f0 2024-03-29T11:25:52.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.570+0000 7feeb679d640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:52.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.570+0000 7feeb679d640 10 monclient(hunting): authenticate will time out at 1212.866180s 2024-03-29T11:25:52.561 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.572+0000 7feeacff9640 10 client.?.objecter ms_handle_connect 0x7feeb00a9d70 2024-03-29T11:25:52.561 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.572+0000 7feeacff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:52.561 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.572+0000 7feeacff9640 10 client.?.objecter ms_handle_connect 0x7feeb01457f0 2024-03-29T11:25:52.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.572+0000 7feeacff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:52.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.572+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699333482 0 0) 0x7feeb0071270 con 0x7feeb01457f0 2024-03-29T11:25:52.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.572+0000 7feeacff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:52.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.573+0000 7feeacff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:52.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.573+0000 7feeacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:52.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.573+0000 7feeacff9640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:25:52.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.573+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee90004eb0 con 0x7feeb01457f0 2024-03-29T11:25:52.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.573+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317245337 0 0) 0x7feeb0070f90 con 0x7feeb00a9d70 2024-03-29T11:25:52.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.573+0000 7feeacff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:52.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.573+0000 7feeacff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:52.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.573+0000 7feeacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:52.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.573+0000 7feeacff9640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:25:52.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.573+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeb0071270 con 0x7feeb00a9d70 2024-03-29T11:25:52.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.573+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2046625551 0 0) 0x7fee90004eb0 con 0x7feeb01457f0 2024-03-29T11:25:52.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.573+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feeb0070f90 con 0x7feeb01457f0 2024-03-29T11:25:52.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.573+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fee98001950 con 0x7feeb01457f0 2024-03-29T11:25:52.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.573+0000 7feeacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:52.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:52.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 10 monclient(hunting): dump: 2024-03-29T11:25:52.563 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:52.563 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:52.563 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:52.564 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:52.564 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:52.564 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:52.564 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:52.564 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:52.564 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:52.564 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:52.564 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:52.564 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 10 client.?.objecter ms_handle_connect 0x7feeb013f650 2024-03-29T11:25:52.564 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:52.564 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 886738053 0 0) 0x7feeb0071270 con 0x7feeb00a9d70 2024-03-29T11:25:52.564 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee90004eb0 con 0x7feeb00a9d70 2024-03-29T11:25:52.564 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135288511 0 0) 0x7feeb01dc1c0 con 0x7feeb013f650 2024-03-29T11:25:52.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:52.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:52.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:52.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:25:52.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeb0071270 con 0x7feeb013f650 2024-03-29T11:25:52.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7feea8001da0 con 0x7feeb00a9d70 2024-03-29T11:25:52.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:52.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:52.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 10 monclient(hunting): dump: 2024-03-29T11:25:52.565 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:52.565 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:52.565 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:52.565 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:52.565 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:52.566 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:52.566 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:52.566 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:52.566 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:52.566 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:52.566 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1318315865 0 0) 0x7fee90004eb0 con 0x7feeb00a9d70 2024-03-29T11:25:52.566 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 >> v1:172.21.15.67:6790/0 conn(0x7feeb01457f0 legacy=0x7feeb006eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:52.566 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 >> v1:172.21.15.67:6789/0 conn(0x7feeb013f650 legacy=0x7feeb00719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:52.566 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 10 monclient: _finish_hunting 0 2024-03-29T11:25:52.566 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 1 monclient: found mon.b 2024-03-29T11:25:52.566 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:52.566 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:25:52.566 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.574+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb01dc650 con 0x7feeb00a9d70 2024-03-29T11:25:52.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeacff9640 10 monclient: _finish_auth 0 2024-03-29T11:25:52.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeacff9640 10 monclient: _check_auth_tickets 2024-03-29T11:25:52.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:52.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeb679d640 5 monclient: authenticate success, global_id 4417 2024-03-29T11:25:52.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeb679d640 10 monclient: _renew_subs 2024-03-29T11:25:52.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeb679d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:25:52.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeb679d640 1 -- 172.21.15.67:0/499693617 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb006f600 con 0x7feeb00a9d70 2024-03-29T11:25:52.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feea8001fe0 con 0x7feeb00a9d70 2024-03-29T11:25:52.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeb679d640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:52.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeacff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:52.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7feea8004fb0 con 0x7feeb00a9d70 2024-03-29T11:25:52.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeacff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:52.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeacff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:52.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeacff9640 10 monclient: dump: 2024-03-29T11:25:52.568 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:52.568 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:52.568 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:52.568 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:52.568 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:52.568 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:52.568 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:52.568 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:52.568 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:52.568 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:52.568 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeb679d640 10 monclient: _renew_subs 2024-03-29T11:25:52.568 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeb679d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:25:52.568 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeb679d640 1 -- 172.21.15.67:0/499693617 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb00b0830 con 0x7feeb00a9d70 2024-03-29T11:25:52.569 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeadffb640 4 set_mon_vals no callback set 2024-03-29T11:25:52.569 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.575+0000 7feeb679d640 1 librados: init done 2024-03-29T11:25:52.569 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.577+0000 7feeb679d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:52.569 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.577+0000 7feeb679d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:52.569 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.577+0000 7feeb679d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:25:52.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.577+0000 7feeb679d640 1 -- 172.21.15.67:0/499693617 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb006f890 con 0x7feeb00a9d70 2024-03-29T11:25:52.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.577+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7feea801af80 con 0x7feeb00a9d70 2024-03-29T11:25:52.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.580+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7feea8084d90 con 0x7feeb00a9d70 2024-03-29T11:25:52.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.580+0000 7feeacff9640 10 client.4417.objecter ms_dispatch 0x7feeb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:52.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.580+0000 7feeacff9640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:52.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.580+0000 7feeacff9640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:52.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.580+0000 7feeacff9640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-29T11:25:52.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.580+0000 7feeaf7fe640 10 monclient: get_auth_request con 0x7fee9006e170 auth_method 0 2024-03-29T11:25:52.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.582+0000 7feeacff9640 10 client.4417.objecter ms_handle_connect 0x7fee9006e170 2024-03-29T11:25:52.572 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.586+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feea8050670 con 0x7feeb00a9d70 2024-03-29T11:25:52.572 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.586+0000 7feeacff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:52.572 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.586+0000 7feeacff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:52.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.914+0000 7feeb679d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:52.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.914+0000 7feeb679d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:52.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.914+0000 7feeb679d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:25:52.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.914+0000 7feeb679d640 1 -- 172.21.15.67:0/499693617 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feeb00720f0 con 0x7feeb00a9d70 2024-03-29T11:25:52.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.914+0000 7feeacff9640 1 -- 172.21.15.67:0/499693617 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7feea80525a0 con 0x7feeb00a9d70 2024-03-29T11:25:52.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.915+0000 7feeacff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:52.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.915+0000 7feeacff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:52.901 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:25:52.901 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:52.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.919+0000 7feeb679d640 10 librados: watch_flush enter 2024-03-29T11:25:52.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.919+0000 7feeb679d640 10 librados: watch_flush exit 2024-03-29T11:25:52.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.919+0000 7feeb679d640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-29T11:25:52.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.919+0000 7feeb679d640 10 client.4417.objecter successfully canceled tick 2024-03-29T11:25:52.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.919+0000 7feeb679d640 1 -- 172.21.15.67:0/499693617 >> v1:172.21.15.67:6816/404787197 conn(0x7fee9006e170 legacy=0x7fee90070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:52.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.919+0000 7feeb679d640 10 monclient: shutdown 2024-03-29T11:25:52.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.919+0000 7feeb679d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:52.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.919+0000 7feeb679d640 1 -- 172.21.15.67:0/499693617 >> v1:172.21.15.82:6789/0 conn(0x7feeb00a9d70 legacy=0x7feeb01db8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:52.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.919+0000 7feeb679d640 1 -- 172.21.15.67:0/499693617 shutdown_connections 2024-03-29T11:25:52.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.919+0000 7feeb679d640 1 -- 172.21.15.67:0/499693617 >> 172.21.15.67:0/499693617 conn(0x7feeb0066a00 msgr2=0x7feeb00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:52.911 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.924+0000 7feeb679d640 1 -- 172.21.15.67:0/499693617 shutdown_connections 2024-03-29T11:25:52.911 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.924+0000 7feeb679d640 1 -- 172.21.15.67:0/499693617 wait complete. 2024-03-29T11:25:52.911 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:52.924+0000 7feeb679d640 1 librados: shutdown 2024-03-29T11:25:52.926 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:59.733 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:59.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.893+0000 7fa73f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:25:59.879 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.893+0000 7fa73f59e640 10 monclient: build_initial_monmap 2024-03-29T11:25:59.879 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.893+0000 7fa73f59e640 10 monclient: monmap: 2024-03-29T11:25:59.879 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:59.879 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:59.879 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:25:59.894350+0000 2024-03-29T11:25:59.879 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:25:59.894350+0000 2024-03-29T11:25:59.879 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:59.879 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:59.879 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:59.879 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:25:59.879 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:25:59.879 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:59.880 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.894+0000 7fa73f59e640 1 Processor -- start 2024-03-29T11:25:59.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.894+0000 7fa73f59e640 1 -- start start 2024-03-29T11:25:59.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.894+0000 7fa73f59e640 10 monclient: init 2024-03-29T11:25:59.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.895+0000 7fa73f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:59.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.895+0000 7fa73f59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:59.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.895+0000 7fa73f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fa738006a70 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:59.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.895+0000 7fa73f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fa73802b100 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:59.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.895+0000 7fa73f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fa738029c30 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:59.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.895+0000 7fa73f59e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73802a270 con 0x7fa738006a70 2024-03-29T11:25:59.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.895+0000 7fa73f59e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73802c440 con 0x7fa73802b100 2024-03-29T11:25:59.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.896+0000 7fa73f59e640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7380c9440 con 0x7fa738029c30 2024-03-29T11:25:59.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.896+0000 7fa73f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:59.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.896+0000 7fa73f59e640 10 monclient(hunting): authenticate will time out at 1220.191910s 2024-03-29T11:25:59.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.896+0000 7fa73e59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa738006a70 0x7fa7380c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58070/0 (socket says 172.21.15.67:58070) 2024-03-29T11:25:59.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.896+0000 7fa73e59c640 1 -- 172.21.15.67:0/181393264 learned_addr learned my addr 172.21.15.67:0/181393264 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:59.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.897+0000 7fa73d59a640 1 -- 172.21.15.67:0/181393264 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764625152 0 0) 0x7fa73802c440 con 0x7fa73802b100 2024-03-29T11:25:59.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.897+0000 7fa73d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:59.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.897+0000 7fa73d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:59.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.897+0000 7fa73d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:59.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.897+0000 7fa73d59a640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:25:59.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.897+0000 7fa73d59a640 1 -- 172.21.15.67:0/181393264 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa728003520 con 0x7fa73802b100 2024-03-29T11:25:59.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.897+0000 7fa73d59a640 1 -- 172.21.15.67:0/181393264 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162184856 0 0) 0x7fa73802a270 con 0x7fa738006a70 2024-03-29T11:25:59.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.897+0000 7fa73d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:59.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.897+0000 7fa73d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:59.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.897+0000 7fa73d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:59.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.897+0000 7fa73d59a640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:25:59.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.897+0000 7fa73d59a640 1 -- 172.21.15.67:0/181393264 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa73802c440 con 0x7fa738006a70 2024-03-29T11:25:59.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.897+0000 7fa73d59a640 1 -- 172.21.15.67:0/181393264 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2737148345 0 0) 0x7fa728003520 con 0x7fa73802b100 2024-03-29T11:25:59.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.897+0000 7fa73d59a640 1 -- 172.21.15.67:0/181393264 >> v1:172.21.15.67:6790/0 conn(0x7fa738029c30 legacy=0x7fa7380d74a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:59.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73d59a640 1 -- 172.21.15.67:0/181393264 >> v1:172.21.15.67:6789/0 conn(0x7fa738006a70 legacy=0x7fa7380c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:25:59.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73d59a640 1 monclient: found mon.noname-c 2024-03-29T11:25:59.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:59.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:25:59.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73d59a640 1 -- 172.21.15.67:0/181393264 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7380c95d0 con 0x7fa73802b100 2024-03-29T11:25:59.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73d59a640 10 monclient: _finish_auth 0 2024-03-29T11:25:59.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:25:59.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:59.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73d59a640 1 -- 172.21.15.67:0/181393264 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa730001690 con 0x7fa73802b100 2024-03-29T11:25:59.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:59.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:59.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73d59a640 10 monclient: dump: 2024-03-29T11:25:59.888 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:59.888 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:59.888 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:59.888 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:59.888 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:59.888 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:59.888 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:59.888 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:59.889 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:59.889 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:59.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73f59e640 5 monclient: authenticate success, global_id 4420 2024-03-29T11:25:59.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.898+0000 7fa73f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:59.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.899+0000 7fa73d59a640 1 -- 172.21.15.67:0/181393264 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7300017f0 con 0x7fa73802b100 2024-03-29T11:25:59.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.899+0000 7fa73d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:59.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.899+0000 7fa73d59a640 1 -- 172.21.15.67:0/181393264 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa730001c00 con 0x7fa73802b100 2024-03-29T11:25:59.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.899+0000 7fa73d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:59.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.899+0000 7fa73d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:59.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.899+0000 7fa73d59a640 10 monclient: dump: 2024-03-29T11:25:59.889 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:59.889 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:59.889 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:59.889 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:59.890 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:59.890 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:59.890 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:59.890 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:59.890 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:59.890 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:59.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.899+0000 7fa73f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:59.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.899+0000 7fa73f59e640 4 set_mon_vals no callback set 2024-03-29T11:25:59.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.900+0000 7fa73f59e640 10 monclient: shutdown 2024-03-29T11:25:59.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.900+0000 7fa73f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:59.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.900+0000 7fa73f59e640 1 -- 172.21.15.67:0/181393264 >> v1:172.21.15.82:6789/0 conn(0x7fa73802b100 legacy=0x7fa7380cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.900+0000 7fa73f59e640 1 -- 172.21.15.67:0/181393264 shutdown_connections 2024-03-29T11:25:59.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.900+0000 7fa73f59e640 1 -- 172.21.15.67:0/181393264 >> 172.21.15.67:0/181393264 conn(0x7fa7380058f0 msgr2=0x7fa738005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:59.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.900+0000 7fa73f59e640 1 -- 172.21.15.67:0/181393264 shutdown_connections 2024-03-29T11:25:59.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.901+0000 7fa73f59e640 1 -- 172.21.15.67:0/181393264 wait complete. 2024-03-29T11:25:59.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.902+0000 7fa73f59e640 10 monclient: build_initial_monmap 2024-03-29T11:25:59.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.902+0000 7fa73f59e640 10 monclient: monmap: 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.903+0000 7fa73f59e640 1 librados: starting msgr at 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.903+0000 7fa73f59e640 1 librados: starting objecter 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.903+0000 7fa73f59e640 1 Processor -- start 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.903+0000 7fa73f59e640 1 -- start start 2024-03-29T11:25:59.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.903+0000 7fa73f59e640 1 librados: setting wanted keys 2024-03-29T11:25:59.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.903+0000 7fa73f59e640 1 librados: calling monclient init 2024-03-29T11:25:59.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.903+0000 7fa73f59e640 10 monclient: init 2024-03-29T11:25:59.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.904+0000 7fa73f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:59.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.904+0000 7fa73f59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:59.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.904+0000 7fa73f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fa738006a70 addr v1:172.21.15.82:6789/0 2024-03-29T11:25:59.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.904+0000 7fa73f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fa738029c30 addr v1:172.21.15.67:6789/0 2024-03-29T11:25:59.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.904+0000 7fa73f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fa7380cd5c0 addr v1:172.21.15.67:6790/0 2024-03-29T11:25:59.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.904+0000 7fa73f59e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7380ceed0 con 0x7fa738029c30 2024-03-29T11:25:59.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.904+0000 7fa73f59e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7380ccdc0 con 0x7fa738006a70 2024-03-29T11:25:59.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.904+0000 7fa73f59e640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7380cd0a0 con 0x7fa7380cd5c0 2024-03-29T11:25:59.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.904+0000 7fa73f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:59.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.905+0000 7fa73e59c640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fa738006a70 0x7fa7380cdce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:49232/0 (socket says 172.21.15.67:49232) 2024-03-29T11:25:59.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.905+0000 7fa73e59c640 1 -- 172.21.15.67:0/4006902846 learned_addr learned my addr 172.21.15.67:0/4006902846 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.905+0000 7fa73f59e640 10 monclient(hunting): authenticate will time out at 1220.201121s 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.905+0000 7fa72effd640 10 client.?.objecter ms_handle_connect 0x7fa738006a70 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.905+0000 7fa72effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.905+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403923633 0 0) 0x7fa7380ccdc0 con 0x7fa738006a70 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.905+0000 7fa72effd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.905+0000 7fa72effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.905+0000 7fa72effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.905+0000 7fa72effd640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.905+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa720004eb0 con 0x7fa738006a70 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.906+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 698013354 0 0) 0x7fa720004eb0 con 0x7fa738006a70 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.906+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7380ccdc0 con 0x7fa738006a70 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.906+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa7340019f0 con 0x7fa738006a70 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.906+0000 7fa72effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.906+0000 7fa72effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:59.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.906+0000 7fa72effd640 10 monclient(hunting): dump: 2024-03-29T11:25:59.902 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:59.902 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:59.902 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:59.902 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:59.902 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:59.902 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:59.902 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:59.902 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:59.902 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:59.903 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:59.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.906+0000 7fa72effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:59.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.907+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1614670703 0 0) 0x7fa7380ccdc0 con 0x7fa738006a70 2024-03-29T11:25:59.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.907+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 >> v1:172.21.15.67:6790/0 conn(0x7fa7380cd5c0 legacy=0x7fa7380bf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.907+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 >> v1:172.21.15.67:6789/0 conn(0x7fa738029c30 legacy=0x7fa7380ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.907+0000 7fa72effd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:59.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.907+0000 7fa72effd640 1 monclient: found mon.b 2024-03-29T11:25:59.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.907+0000 7fa72effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:59.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.907+0000 7fa72effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:25:59.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.907+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7380c0430 con 0x7fa738006a70 2024-03-29T11:25:59.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.907+0000 7fa72effd640 10 monclient: _finish_auth 0 2024-03-29T11:25:59.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.907+0000 7fa72effd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:59.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.907+0000 7fa72effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:59.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.907+0000 7fa73f59e640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:25:59.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.907+0000 7fa73f59e640 10 monclient: _renew_subs 2024-03-29T11:25:59.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.908+0000 7fa73f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:25:59.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.908+0000 7fa73f59e640 1 -- 172.21.15.67:0/4006902846 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7380cae00 con 0x7fa738006a70 2024-03-29T11:25:59.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.908+0000 7fa73f59e640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:59.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.908+0000 7fa73f59e640 10 monclient: _renew_subs 2024-03-29T11:25:59.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.908+0000 7fa73f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:25:59.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.908+0000 7fa73f59e640 1 -- 172.21.15.67:0/4006902846 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7380c0170 con 0x7fa738006a70 2024-03-29T11:25:59.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.908+0000 7fa73f59e640 1 librados: init done 2024-03-29T11:25:59.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.908+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa734001b50 con 0x7fa738006a70 2024-03-29T11:25:59.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.908+0000 7fa72effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:59.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.908+0000 7fa73cd99640 4 set_mon_vals no callback set 2024-03-29T11:25:59.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.909+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa734004c30 con 0x7fa738006a70 2024-03-29T11:25:59.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.909+0000 7fa72effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:59.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.909+0000 7fa72effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:59.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.909+0000 7fa72effd640 10 monclient: dump: 2024-03-29T11:25:59.905 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:25:59.905 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:25:59.905 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:59.905 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:25:59.905 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:59.905 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:25:59.905 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:25:59.905 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:25:59.905 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:25:59.905 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:25:59.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.910+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fa734004d90 con 0x7fa738006a70 2024-03-29T11:25:59.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.911+0000 7fa73dd9b640 10 monclient: get_auth_request con 0x7fa72006dc20 auth_method 0 2024-03-29T11:25:59.905 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.911+0000 7fa73f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:59.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.911+0000 7fa73f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:59.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.911+0000 7fa73f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:25:59.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.911+0000 7fa73f59e640 1 -- 172.21.15.67:0/4006902846 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa724005080 con 0x7fa738006a70 2024-03-29T11:25:59.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.912+0000 7fa72effd640 10 client.4423.objecter ms_handle_connect 0x7fa72006dc20 2024-03-29T11:25:59.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.912+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fa734084be0 con 0x7fa738006a70 2024-03-29T11:25:59.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.912+0000 7fa72effd640 10 client.4423.objecter ms_dispatch 0x7fa740000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:59.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.912+0000 7fa72effd640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:59.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.912+0000 7fa72effd640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:59.906 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.912+0000 7fa72effd640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T11:25:59.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.918+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7340504c0 con 0x7fa738006a70 2024-03-29T11:25:59.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.918+0000 7fa72effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:59.907 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:25:59.918+0000 7fa72effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:00.251 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.266+0000 7fa73f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.266+0000 7fa73f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.266+0000 7fa73f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:26:00.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.266+0000 7fa73f59e640 1 -- 172.21.15.67:0/4006902846 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa724005330 con 0x7fa738006a70 2024-03-29T11:26:00.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.267+0000 7fa72effd640 1 -- 172.21.15.67:0/4006902846 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fa7340523f0 con 0x7fa738006a70 2024-03-29T11:26:00.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.267+0000 7fa72effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.267+0000 7fa72effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:00.256 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:26:00.256 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:00.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.274+0000 7fa73f59e640 10 librados: watch_flush enter 2024-03-29T11:26:00.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.274+0000 7fa73f59e640 10 librados: watch_flush exit 2024-03-29T11:26:00.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.274+0000 7fa73f59e640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T11:26:00.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.274+0000 7fa73f59e640 10 client.4423.objecter successfully canceled tick 2024-03-29T11:26:00.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.274+0000 7fa73f59e640 1 -- 172.21.15.67:0/4006902846 >> v1:172.21.15.67:6816/404787197 conn(0x7fa72006dc20 legacy=0x7fa7200700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.274+0000 7fa73f59e640 10 monclient: shutdown 2024-03-29T11:26:00.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.274+0000 7fa73f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:00.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.274+0000 7fa73f59e640 1 -- 172.21.15.67:0/4006902846 >> v1:172.21.15.82:6789/0 conn(0x7fa738006a70 legacy=0x7fa7380cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.275+0000 7fa73f59e640 1 -- 172.21.15.67:0/4006902846 shutdown_connections 2024-03-29T11:26:00.263 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.275+0000 7fa73f59e640 1 -- 172.21.15.67:0/4006902846 >> 172.21.15.67:0/4006902846 conn(0x7fa7380058f0 msgr2=0x7fa738005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:00.263 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.275+0000 7fa73f59e640 1 -- 172.21.15.67:0/4006902846 shutdown_connections 2024-03-29T11:26:00.263 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.275+0000 7fa73f59e640 1 -- 172.21.15.67:0/4006902846 wait complete. 2024-03-29T11:26:00.264 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:00.277+0000 7fa73f59e640 1 librados: shutdown 2024-03-29T11:26:00.283 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:07.090 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:07.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.260+0000 7f0edab32640 10 monclient: get_monmap_and_config 2024-03-29T11:26:07.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.260+0000 7f0edab32640 10 monclient: build_initial_monmap 2024-03-29T11:26:07.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.260+0000 7f0edab32640 10 monclient: monmap: 2024-03-29T11:26:07.245 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:07.245 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:07.245 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:26:07.260816+0000 2024-03-29T11:26:07.246 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:26:07.260816+0000 2024-03-29T11:26:07.246 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:07.246 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:07.246 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:07.246 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:26:07.246 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:26:07.246 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:07.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.261+0000 7f0edab32640 1 Processor -- start 2024-03-29T11:26:07.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.261+0000 7f0edab32640 1 -- start start 2024-03-29T11:26:07.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.261+0000 7f0edab32640 10 monclient: init 2024-03-29T11:26:07.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.262+0000 7f0edab32640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:07.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.262+0000 7f0edab32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:07.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.262+0000 7f0edab32640 10 monclient(hunting): picked mon.noname-a con 0x7f0ed41457f0 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:07.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.262+0000 7f0edab32640 10 monclient(hunting): picked mon.noname-b con 0x7f0ed413f650 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:07.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.262+0000 7f0edab32640 10 monclient(hunting): picked mon.noname-c con 0x7f0ed40a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:07.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.262+0000 7f0edab32640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed40aaef0 con 0x7f0ed41457f0 2024-03-29T11:26:07.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.262+0000 7f0edab32640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed4145bd0 con 0x7f0ed40a9d70 2024-03-29T11:26:07.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.262+0000 7f0ed3fff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f0ed413f650 0x7f0ed40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:46238/0 (socket says 172.21.15.67:46238) 2024-03-29T11:26:07.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.262+0000 7f0ed3fff640 1 -- 172.21.15.67:0/3400363474 learned_addr learned my addr 172.21.15.67:0/3400363474 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:07.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.262+0000 7f0edab32640 1 -- 172.21.15.67:0/3400363474 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed4145eb0 con 0x7f0ed413f650 2024-03-29T11:26:07.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.262+0000 7f0edab32640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:07.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.262+0000 7f0edab32640 10 monclient(hunting): authenticate will time out at 1227.558598s 2024-03-29T11:26:07.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.263+0000 7f0ed37fe640 1 -- 172.21.15.67:0/3400363474 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532159876 0 0) 0x7f0ed4145eb0 con 0x7f0ed413f650 2024-03-29T11:26:07.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.263+0000 7f0ed37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.263+0000 7f0ed37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.263+0000 7f0ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.263+0000 7f0ed37fe640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:26:07.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.263+0000 7f0ed37fe640 1 -- 172.21.15.67:0/3400363474 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ec4003520 con 0x7f0ed413f650 2024-03-29T11:26:07.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.263+0000 7f0ed37fe640 1 -- 172.21.15.67:0/3400363474 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243149914 0 0) 0x7f0ed40aaef0 con 0x7f0ed41457f0 2024-03-29T11:26:07.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.263+0000 7f0ed37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.263+0000 7f0ed37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.263+0000 7f0ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.263+0000 7f0ed37fe640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:26:07.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.263+0000 7f0ed37fe640 1 -- 172.21.15.67:0/3400363474 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ed4145eb0 con 0x7f0ed41457f0 2024-03-29T11:26:07.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 1 -- 172.21.15.67:0/3400363474 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3491101484 0 0) 0x7f0ec4003520 con 0x7f0ed413f650 2024-03-29T11:26:07.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 1 -- 172.21.15.67:0/3400363474 >> v1:172.21.15.82:6789/0 conn(0x7f0ed40a9d70 legacy=0x7f0ed414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 1 -- 172.21.15.67:0/3400363474 >> v1:172.21.15.67:6789/0 conn(0x7f0ed41457f0 legacy=0x7f0ed413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:07.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 1 monclient: found mon.noname-b 2024-03-29T11:26:07.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:07.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:26:07.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 1 -- 172.21.15.67:0/3400363474 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed4146340 con 0x7f0ed413f650 2024-03-29T11:26:07.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:07.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:07.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:07.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 1 -- 172.21.15.67:0/3400363474 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0ec0001690 con 0x7f0ed413f650 2024-03-29T11:26:07.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0edab32640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:26:07.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0edab32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:07.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:07.255 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 10 monclient: dump: 2024-03-29T11:26:07.255 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:07.255 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:07.255 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:07.255 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:07.256 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.256 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:07.256 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:07.256 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:07.256 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:07.256 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:07.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 1 -- 172.21.15.67:0/3400363474 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ec00018b0 con 0x7f0ed413f650 2024-03-29T11:26:07.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:07.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 1 -- 172.21.15.67:0/3400363474 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0ec0001c40 con 0x7f0ed413f650 2024-03-29T11:26:07.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.264+0000 7f0ed37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:07.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.265+0000 7f0ed37fe640 10 monclient: dump: 2024-03-29T11:26:07.256 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:07.256 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:07.257 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:07.257 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:07.257 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.257 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:07.257 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:07.257 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:07.257 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:07.257 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:07.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.265+0000 7f0edab32640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:07.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.265+0000 7f0edab32640 4 set_mon_vals no callback set 2024-03-29T11:26:07.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.265+0000 7f0edab32640 10 monclient: shutdown 2024-03-29T11:26:07.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.266+0000 7f0edab32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:07.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.266+0000 7f0edab32640 1 -- 172.21.15.67:0/3400363474 >> v1:172.21.15.67:6790/0 conn(0x7f0ed413f650 legacy=0x7f0ed40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.266+0000 7f0edab32640 1 -- 172.21.15.67:0/3400363474 shutdown_connections 2024-03-29T11:26:07.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.266+0000 7f0edab32640 1 -- 172.21.15.67:0/3400363474 >> 172.21.15.67:0/3400363474 conn(0x7f0ed4066a00 msgr2=0x7f0ed4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:07.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.266+0000 7f0edab32640 1 -- 172.21.15.67:0/3400363474 shutdown_connections 2024-03-29T11:26:07.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.266+0000 7f0edab32640 1 -- 172.21.15.67:0/3400363474 wait complete. 2024-03-29T11:26:07.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.267+0000 7f0edab32640 10 monclient: build_initial_monmap 2024-03-29T11:26:07.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.267+0000 7f0edab32640 10 monclient: monmap: 2024-03-29T11:26:07.258 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:07.258 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:07.258 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:26:07.258 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:26:07.258 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:07.258 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:07.258 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:07.258 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:26:07.259 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:26:07.259 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:07.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.268+0000 7f0edab32640 1 librados: starting msgr at 2024-03-29T11:26:07.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.268+0000 7f0edab32640 1 librados: starting objecter 2024-03-29T11:26:07.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.268+0000 7f0edab32640 1 Processor -- start 2024-03-29T11:26:07.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.268+0000 7f0edab32640 1 -- start start 2024-03-29T11:26:07.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.268+0000 7f0edab32640 1 librados: setting wanted keys 2024-03-29T11:26:07.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.268+0000 7f0edab32640 1 librados: calling monclient init 2024-03-29T11:26:07.259 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.268+0000 7f0edab32640 10 monclient: init 2024-03-29T11:26:07.268 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.269+0000 7f0edab32640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:07.268 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.269+0000 7f0edab32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:07.268 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.269+0000 7f0edab32640 10 monclient(hunting): picked mon.noname-c con 0x7f0ed40a9d70 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:07.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.269+0000 7f0ed88ce640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f0ed40a9d70 0x7f0ed406f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:46654/0 (socket says 172.21.15.67:46654) 2024-03-29T11:26:07.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.269+0000 7f0ed88ce640 1 -- 172.21.15.67:0/2185950788 learned_addr learned my addr 172.21.15.67:0/2185950788 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:07.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.269+0000 7f0edab32640 10 monclient(hunting): picked mon.noname-b con 0x7f0ed413f650 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:07.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.270+0000 7f0edab32640 10 monclient(hunting): picked mon.noname-a con 0x7f0ed41457f0 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:07.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.270+0000 7f0ed17fa640 10 client.?.objecter ms_handle_connect 0x7f0ed40a9d70 2024-03-29T11:26:07.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.270+0000 7f0ed17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:07.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.270+0000 7f0ed17fa640 10 client.?.objecter ms_handle_connect 0x7f0ed413f650 2024-03-29T11:26:07.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.270+0000 7f0ed17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:07.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.270+0000 7f0edab32640 1 -- 172.21.15.67:0/2185950788 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed406efa0 con 0x7f0ed41457f0 2024-03-29T11:26:07.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.270+0000 7f0ed17fa640 10 client.?.objecter ms_handle_connect 0x7f0ed41457f0 2024-03-29T11:26:07.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.270+0000 7f0ed17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:07.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.270+0000 7f0edab32640 1 -- 172.21.15.67:0/2185950788 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed406f280 con 0x7f0ed413f650 2024-03-29T11:26:07.269 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.270+0000 7f0edab32640 1 -- 172.21.15.67:0/2185950788 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed406f560 con 0x7f0ed40a9d70 2024-03-29T11:26:07.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.270+0000 7f0edab32640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:07.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.271+0000 7f0edab32640 10 monclient(hunting): authenticate will time out at 1227.566959s 2024-03-29T11:26:07.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.271+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975611910 0 0) 0x7f0ed406f280 con 0x7f0ed413f650 2024-03-29T11:26:07.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.271+0000 7f0ed17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.271+0000 7f0ed17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.271+0000 7f0ed17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.271+0000 7f0ed17fa640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:26:07.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.271+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eb8004eb0 con 0x7f0ed413f650 2024-03-29T11:26:07.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.271+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898969848 0 0) 0x7f0ed406efa0 con 0x7f0ed41457f0 2024-03-29T11:26:07.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.271+0000 7f0ed17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.272+0000 7f0ed17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.272+0000 7f0ed17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.270 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.272+0000 7f0ed17fa640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:26:07.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.272+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ed406f280 con 0x7f0ed41457f0 2024-03-29T11:26:07.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.272+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901188436 0 0) 0x7f0ed406f560 con 0x7f0ed40a9d70 2024-03-29T11:26:07.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.272+0000 7f0ed17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.272+0000 7f0ed17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.272+0000 7f0ed17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.272+0000 7f0ed17fa640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:26:07.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.272+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ed406efa0 con 0x7f0ed40a9d70 2024-03-29T11:26:07.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.272+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 749132162 0 0) 0x7f0eb8004eb0 con 0x7f0ed413f650 2024-03-29T11:26:07.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.272+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ed406f560 con 0x7f0ed413f650 2024-03-29T11:26:07.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.272+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0ec0001610 con 0x7f0ed413f650 2024-03-29T11:26:07.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.272+0000 7f0ed17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.273+0000 7f0ed17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:07.271 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.273+0000 7f0ed17fa640 10 monclient(hunting): dump: 2024-03-29T11:26:07.272 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:07.272 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:07.272 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:07.272 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:07.272 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.272 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:07.272 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:07.272 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:07.272 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:07.272 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:07.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.273+0000 7f0ed17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:07.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.273+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2146508208 0 0) 0x7f0ed406f280 con 0x7f0ed41457f0 2024-03-29T11:26:07.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.273+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0eb8004eb0 con 0x7f0ed41457f0 2024-03-29T11:26:07.272 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.273+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2934073489 0 0) 0x7f0ed406efa0 con 0x7f0ed40a9d70 2024-03-29T11:26:07.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.273+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ed406f280 con 0x7f0ed40a9d70 2024-03-29T11:26:07.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.273+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0ecc001b20 con 0x7f0ed41457f0 2024-03-29T11:26:07.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.273+0000 7f0ed17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.273+0000 7f0ed17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:07.273 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.273+0000 7f0ed17fa640 10 monclient(hunting): dump: 2024-03-29T11:26:07.273 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:07.273 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:07.273 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:07.273 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:07.273 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.273 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:07.273 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:07.273 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:07.273 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:07.274 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:07.274 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.273+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0ec80017d0 con 0x7f0ed40a9d70 2024-03-29T11:26:07.274 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.274+0000 7f0ed17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.274 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.274+0000 7f0ed17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:07.274 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.274+0000 7f0ed17fa640 10 monclient(hunting): dump: 2024-03-29T11:26:07.274 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:07.274 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:07.274 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:07.274 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:07.274 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.274 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:07.274 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:07.275 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:07.275 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:07.275 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:07.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.274+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2125692524 0 0) 0x7f0eb8004eb0 con 0x7f0ed41457f0 2024-03-29T11:26:07.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.274+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 >> v1:172.21.15.67:6790/0 conn(0x7f0ed40a9d70 legacy=0x7f0ed406f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.274+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 >> v1:172.21.15.82:6789/0 conn(0x7f0ed413f650 legacy=0x7f0ed41df1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.274+0000 7f0ed17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:26:07.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.274+0000 7f0ed17fa640 1 monclient: found mon.a 2024-03-29T11:26:07.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.274+0000 7f0ed17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:07.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.274+0000 7f0ed17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:07.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.274+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed4145120 con 0x7f0ed41457f0 2024-03-29T11:26:07.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0ed17fa640 10 monclient: _finish_auth 0 2024-03-29T11:26:07.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0ed17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:26:07.275 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0ed17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:07.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0edab32640 5 monclient: authenticate success, global_id 4458 2024-03-29T11:26:07.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0edab32640 10 monclient: _renew_subs 2024-03-29T11:26:07.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0edab32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:07.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0edab32640 1 -- 172.21.15.67:0/2185950788 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed41dffc0 con 0x7f0ed41457f0 2024-03-29T11:26:07.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0edab32640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:07.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0edab32640 10 monclient: _renew_subs 2024-03-29T11:26:07.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0edab32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:07.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0edab32640 1 -- 172.21.15.67:0/2185950788 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed4144d50 con 0x7f0ed41457f0 2024-03-29T11:26:07.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0edab32640 1 librados: init done 2024-03-29T11:26:07.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ecc002090 con 0x7f0ed41457f0 2024-03-29T11:26:07.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0ed17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:07.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0ed27fc640 4 set_mon_vals no callback set 2024-03-29T11:26:07.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.275+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0ecc003b10 con 0x7f0ed41457f0 2024-03-29T11:26:07.276 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.276+0000 7f0ed17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.276+0000 7f0ed17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:07.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.276+0000 7f0ed17fa640 10 monclient: dump: 2024-03-29T11:26:07.277 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:07.277 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:07.277 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:07.277 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:07.277 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.277 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:07.277 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:07.277 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:07.277 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:07.277 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:07.277 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.277+0000 7f0edab32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.277+0000 7f0edab32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.277+0000 7f0edab32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:07.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.277+0000 7f0edab32640 1 -- 172.21.15.67:0/2185950788 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed4075f10 con 0x7f0ed41457f0 2024-03-29T11:26:07.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.278+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f0ecc019ae0 con 0x7f0ed41457f0 2024-03-29T11:26:07.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.279+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f0ecc083900 con 0x7f0ed41457f0 2024-03-29T11:26:07.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.279+0000 7f0ed88ce640 10 monclient: get_auth_request con 0x7f0eb806e2c0 auth_method 0 2024-03-29T11:26:07.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.280+0000 7f0ed17fa640 10 client.4458.objecter ms_dispatch 0x7f0ed4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:07.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.280+0000 7f0ed17fa640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:07.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.280+0000 7f0ed17fa640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:07.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.280+0000 7f0ed17fa640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-29T11:26:07.278 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.280+0000 7f0ed17fa640 10 client.4458.objecter ms_handle_connect 0x7f0eb806e2c0 2024-03-29T11:26:07.279 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.291+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ecc04f1e0 con 0x7f0ed41457f0 2024-03-29T11:26:07.279 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.291+0000 7f0ed17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.279 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.291+0000 7f0ed17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:07.641 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.656+0000 7f0edab32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:07.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.656+0000 7f0edab32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:07.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.656+0000 7f0edab32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:07.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.656+0000 7f0edab32640 1 -- 172.21.15.67:0/2185950788 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ed40a97f0 con 0x7f0ed41457f0 2024-03-29T11:26:07.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.657+0000 7f0ed17fa640 1 -- 172.21.15.67:0/2185950788 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f0ecc051110 con 0x7f0ed41457f0 2024-03-29T11:26:07.643 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:26:07.643 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:07.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.657+0000 7f0ed17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:07.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.657+0000 7f0ed17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:07.646 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.661+0000 7f0edab32640 10 librados: watch_flush enter 2024-03-29T11:26:07.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.662+0000 7f0edab32640 10 librados: watch_flush exit 2024-03-29T11:26:07.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.662+0000 7f0edab32640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-29T11:26:07.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.662+0000 7f0edab32640 10 client.4458.objecter successfully canceled tick 2024-03-29T11:26:07.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.662+0000 7f0edab32640 1 -- 172.21.15.67:0/2185950788 >> v1:172.21.15.67:6816/404787197 conn(0x7f0eb806e2c0 legacy=0x7f0eb8070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.662+0000 7f0edab32640 10 monclient: shutdown 2024-03-29T11:26:07.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.662+0000 7f0edab32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:07.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.662+0000 7f0edab32640 1 -- 172.21.15.67:0/2185950788 >> v1:172.21.15.67:6789/0 conn(0x7f0ed41457f0 legacy=0x7f0ed41df8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.662+0000 7f0edab32640 1 -- 172.21.15.67:0/2185950788 shutdown_connections 2024-03-29T11:26:07.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.662+0000 7f0edab32640 1 -- 172.21.15.67:0/2185950788 >> 172.21.15.67:0/2185950788 conn(0x7f0ed4066a00 msgr2=0x7f0ed413aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:07.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.663+0000 7f0edab32640 1 -- 172.21.15.67:0/2185950788 shutdown_connections 2024-03-29T11:26:07.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.663+0000 7f0edab32640 1 -- 172.21.15.67:0/2185950788 wait complete. 2024-03-29T11:26:07.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:07.663+0000 7f0edab32640 1 librados: shutdown 2024-03-29T11:26:07.664 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:14.472 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:14.619 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.632+0000 7fcb7b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:26:14.619 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.632+0000 7fcb7b59e640 10 monclient: build_initial_monmap 2024-03-29T11:26:14.619 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.632+0000 7fcb7b59e640 10 monclient: monmap: 2024-03-29T11:26:14.619 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:14.619 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:14.619 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:26:14.633708+0000 2024-03-29T11:26:14.620 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:26:14.633708+0000 2024-03-29T11:26:14.620 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:14.620 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:14.620 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:14.620 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:26:14.620 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:26:14.620 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:14.620 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.634+0000 7fcb7b59e640 1 Processor -- start 2024-03-29T11:26:14.622 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.634+0000 7fcb7b59e640 1 -- start start 2024-03-29T11:26:14.622 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.634+0000 7fcb7b59e640 10 monclient: init 2024-03-29T11:26:14.622 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.635+0000 7fcb7b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:14.622 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.635+0000 7fcb7b59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:14.622 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.635+0000 7fcb7b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fcb7c0a9750 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:14.622 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.635+0000 7fcb7b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fcb7c0ae220 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:14.623 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.635+0000 7fcb7b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fcb7c13b9d0 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:14.623 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.635+0000 7fcb7b59e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c142490 con 0x7fcb7c13b9d0 2024-03-29T11:26:14.623 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.635+0000 7fcb7b59e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c0ac130 con 0x7fcb7c0ae220 2024-03-29T11:26:14.623 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.635+0000 7fcb7b59e640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c0ac410 con 0x7fcb7c0a9750 2024-03-29T11:26:14.623 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.635+0000 7fcb7b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:14.623 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.635+0000 7fcb7b59e640 10 monclient(hunting): authenticate will time out at 1234.931156s 2024-03-29T11:26:14.623 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.635+0000 7fcb7ad9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcb7c13b9d0 0x7fcb7c149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37514/0 (socket says 172.21.15.67:37514) 2024-03-29T11:26:14.623 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.635+0000 7fcb7ad9d640 1 -- 172.21.15.67:0/2236345003 learned_addr learned my addr 172.21.15.67:0/2236345003 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:14.625 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.639+0000 7fcb7959a640 1 -- 172.21.15.67:0/2236345003 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098009387 0 0) 0x7fcb7c142490 con 0x7fcb7c13b9d0 2024-03-29T11:26:14.625 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.639+0000 7fcb7959a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:14.625 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.639+0000 7fcb7959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:14.625 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.639+0000 7fcb7959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:14.625 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.639+0000 7fcb7959a640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:26:14.625 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.639+0000 7fcb7959a640 1 -- 172.21.15.67:0/2236345003 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb64003520 con 0x7fcb7c13b9d0 2024-03-29T11:26:14.625 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.639+0000 7fcb7959a640 1 -- 172.21.15.67:0/2236345003 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651314625 0 0) 0x7fcb7c0ac410 con 0x7fcb7c0a9750 2024-03-29T11:26:14.625 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.639+0000 7fcb7959a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:14.625 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.639+0000 7fcb7959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:14.625 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.639+0000 7fcb7959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:14.625 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.639+0000 7fcb7959a640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:26:14.625 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.639+0000 7fcb7959a640 1 -- 172.21.15.67:0/2236345003 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb7c142490 con 0x7fcb7c0a9750 2024-03-29T11:26:14.628 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7959a640 1 -- 172.21.15.67:0/2236345003 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3206341934 0 0) 0x7fcb7c142490 con 0x7fcb7c0a9750 2024-03-29T11:26:14.628 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7959a640 1 -- 172.21.15.67:0/2236345003 >> v1:172.21.15.82:6789/0 conn(0x7fcb7c0ae220 legacy=0x7fcb7c13b2c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7959a640 1 -- 172.21.15.67:0/2236345003 >> v1:172.21.15.67:6789/0 conn(0x7fcb7c13b9d0 legacy=0x7fcb7c149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7959a640 10 monclient: _finish_hunting 0 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7959a640 1 monclient: found mon.noname-b 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7959a640 1 -- 172.21.15.67:0/2236345003 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c0afa40 con 0x7fcb7c0a9750 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7959a640 10 monclient: _finish_auth 0 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7959a640 10 monclient: _check_auth_tickets 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7b59e640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7959a640 1 -- 172.21.15.67:0/2236345003 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fcb740019f0 con 0x7fcb7c0a9750 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.640+0000 7fcb7959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.641+0000 7fcb7959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:14.629 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.641+0000 7fcb7959a640 10 monclient: dump: 2024-03-29T11:26:14.630 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:14.630 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:14.630 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:14.630 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:14.630 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:14.630 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:14.630 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:14.630 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:14.630 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:14.630 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:14.630 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.641+0000 7fcb7b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:14.630 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.641+0000 7fcb7959a640 1 -- 172.21.15.67:0/2236345003 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb74001c10 con 0x7fcb7c0a9750 2024-03-29T11:26:14.630 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.641+0000 7fcb7959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:14.630 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.641+0000 7fcb7959a640 1 -- 172.21.15.67:0/2236345003 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fcb74001f20 con 0x7fcb7c0a9750 2024-03-29T11:26:14.631 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.641+0000 7fcb7959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:14.631 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.641+0000 7fcb7959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:14.631 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.641+0000 7fcb7959a640 10 monclient: dump: 2024-03-29T11:26:14.631 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:14.631 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:14.631 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:14.631 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:14.631 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:14.631 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:14.631 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:14.631 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:14.631 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:14.631 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:14.631 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.641+0000 7fcb7b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:14.632 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.641+0000 7fcb7b59e640 4 set_mon_vals no callback set 2024-03-29T11:26:14.632 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.642+0000 7fcb7b59e640 10 monclient: shutdown 2024-03-29T11:26:14.632 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.642+0000 7fcb7b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:14.632 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.642+0000 7fcb7b59e640 1 -- 172.21.15.67:0/2236345003 >> v1:172.21.15.67:6790/0 conn(0x7fcb7c0a9750 legacy=0x7fcb7c141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:14.632 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.642+0000 7fcb7b59e640 1 -- 172.21.15.67:0/2236345003 shutdown_connections 2024-03-29T11:26:14.632 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.642+0000 7fcb7b59e640 1 -- 172.21.15.67:0/2236345003 >> 172.21.15.67:0/2236345003 conn(0x7fcb7c0673e0 msgr2=0x7fcb7c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:14.632 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.642+0000 7fcb7b59e640 1 -- 172.21.15.67:0/2236345003 shutdown_connections 2024-03-29T11:26:14.634 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.642+0000 7fcb7b59e640 1 -- 172.21.15.67:0/2236345003 wait complete. 2024-03-29T11:26:14.634 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.643+0000 7fcb7b59e640 10 monclient: build_initial_monmap 2024-03-29T11:26:14.634 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.643+0000 7fcb7b59e640 10 monclient: monmap: 2024-03-29T11:26:14.635 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:14.635 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:14.635 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:26:14.635 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:26:14.635 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:14.635 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:14.635 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:14.635 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:26:14.635 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:26:14.635 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:14.635 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.644+0000 7fcb7b59e640 1 librados: starting msgr at 2024-03-29T11:26:14.635 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.644+0000 7fcb7b59e640 1 librados: starting objecter 2024-03-29T11:26:14.635 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.644+0000 7fcb7b59e640 1 Processor -- start 2024-03-29T11:26:14.635 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.644+0000 7fcb7b59e640 1 -- start start 2024-03-29T11:26:14.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.644+0000 7fcb7b59e640 1 librados: setting wanted keys 2024-03-29T11:26:14.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.644+0000 7fcb7b59e640 1 librados: calling monclient init 2024-03-29T11:26:14.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.644+0000 7fcb7b59e640 10 monclient: init 2024-03-29T11:26:14.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.645+0000 7fcb7b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:14.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.645+0000 7fcb7b59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:14.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.645+0000 7fcb7b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fcb7c0a9750 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:14.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.646+0000 7fcb7b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fcb7c0ae220 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:14.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.646+0000 7fcb7b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fcb7c13b9d0 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:14.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.646+0000 7fcb7b59e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c1ccbb0 con 0x7fcb7c0ae220 2024-03-29T11:26:14.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.646+0000 7fcb7b59e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c1cce90 con 0x7fcb7c13b9d0 2024-03-29T11:26:14.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.646+0000 7fcb7b59e640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c1cbae0 con 0x7fcb7c0a9750 2024-03-29T11:26:14.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.646+0000 7fcb7b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:14.636 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.647+0000 7fcb7b59e640 10 monclient(hunting): authenticate will time out at 1234.943768s 2024-03-29T11:26:14.641 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.650+0000 7fcb7a59c640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fcb7c0a9750 0x7fcb7c1c8170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:43783/0 (socket says 172.21.15.67:43783) 2024-03-29T11:26:14.641 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.650+0000 7fcb7a59c640 1 -- 172.21.15.67:0/519326607 learned_addr learned my addr 172.21.15.67:0/519326607 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:14.641 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.650+0000 7fcb6affd640 10 client.?.objecter ms_handle_connect 0x7fcb7c0a9750 2024-03-29T11:26:14.641 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.650+0000 7fcb6affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:14.641 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.651+0000 7fcb6affd640 10 client.?.objecter ms_handle_connect 0x7fcb7c0ae220 2024-03-29T11:26:14.641 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.651+0000 7fcb6affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:14.641 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.651+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757863512 0 0) 0x7fcb7c1cbae0 con 0x7fcb7c0a9750 2024-03-29T11:26:14.641 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.652+0000 7fcb6affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:14.641 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.652+0000 7fcb6affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:14.641 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.652+0000 7fcb6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:14.641 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.652+0000 7fcb6affd640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:26:14.641 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.652+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb5c004eb0 con 0x7fcb7c0a9750 2024-03-29T11:26:14.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.652+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900716308 0 0) 0x7fcb7c1ccbb0 con 0x7fcb7c0ae220 2024-03-29T11:26:14.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.652+0000 7fcb6affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:14.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.652+0000 7fcb6affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:14.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.652+0000 7fcb6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:14.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.652+0000 7fcb6affd640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:26:14.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.652+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb7c1cbae0 con 0x7fcb7c0ae220 2024-03-29T11:26:14.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.652+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2629790107 0 0) 0x7fcb5c004eb0 con 0x7fcb7c0a9750 2024-03-29T11:26:14.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.653+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb7c1ccbb0 con 0x7fcb7c0a9750 2024-03-29T11:26:14.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.653+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4119478375 0 0) 0x7fcb7c1cbae0 con 0x7fcb7c0ae220 2024-03-29T11:26:14.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.653+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb5c004eb0 con 0x7fcb7c0ae220 2024-03-29T11:26:14.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.653+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fcb74001fa0 con 0x7fcb7c0a9750 2024-03-29T11:26:14.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.653+0000 7fcb6affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:14.642 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.653+0000 7fcb6affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.653+0000 7fcb6affd640 10 monclient(hunting): dump: 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.653+0000 7fcb6affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.653+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fcb6c001730 con 0x7fcb7c0ae220 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.653+0000 7fcb6affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:14.643 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.654+0000 7fcb6affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.654+0000 7fcb6affd640 10 monclient(hunting): dump: 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.654+0000 7fcb6affd640 10 client.?.objecter ms_handle_connect 0x7fcb7c13b9d0 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.654+0000 7fcb6affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.654+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4261251976 0 0) 0x7fcb7c1ccbb0 con 0x7fcb7c0a9750 2024-03-29T11:26:14.644 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.654+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 >> v1:172.21.15.82:6789/0 conn(0x7fcb7c13b9d0 legacy=0x7fcb7c1ee5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:14.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.654+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 >> v1:172.21.15.67:6789/0 conn(0x7fcb7c0ae220 legacy=0x7fcb7c1c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:14.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.654+0000 7fcb6affd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:14.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.654+0000 7fcb6affd640 1 monclient: found mon.c 2024-03-29T11:26:14.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.654+0000 7fcb6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:14.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.654+0000 7fcb6affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:14.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.654+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c1cbc70 con 0x7fcb7c0a9750 2024-03-29T11:26:14.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb6affd640 10 monclient: _finish_auth 0 2024-03-29T11:26:14.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb6affd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:14.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:14.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb7b59e640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:26:14.645 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb7b59e640 10 monclient: _renew_subs 2024-03-29T11:26:14.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb7b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:14.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb7b59e640 1 -- 172.21.15.67:0/519326607 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7c1cd2b0 con 0x7fcb7c0a9750 2024-03-29T11:26:14.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb7b59e640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:14.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb7b59e640 10 monclient: _renew_subs 2024-03-29T11:26:14.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb7b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:14.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb7b59e640 1 -- 172.21.15.67:0/519326607 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7c1cb560 con 0x7fcb7c0a9750 2024-03-29T11:26:14.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb7b59e640 1 librados: init done 2024-03-29T11:26:14.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb74003d40 con 0x7fcb7c0a9750 2024-03-29T11:26:14.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:14.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb6bfff640 4 set_mon_vals no callback set 2024-03-29T11:26:14.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fcb74004090 con 0x7fcb7c0a9750 2024-03-29T11:26:14.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:14.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:14.647 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.657+0000 7fcb7b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:14.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.655+0000 7fcb6affd640 10 monclient: dump: 2024-03-29T11:26:14.648 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:14.648 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:14.648 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:14.648 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:14.648 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:14.648 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:14.648 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:14.648 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:14.648 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:14.648 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:14.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.658+0000 7fcb7b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:14.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.658+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fcb740041f0 con 0x7fcb7c0a9750 2024-03-29T11:26:14.648 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.658+0000 7fcb7b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:14.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.658+0000 7fcb7b59e640 1 -- 172.21.15.67:0/519326607 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb60005080 con 0x7fcb7c0a9750 2024-03-29T11:26:14.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.659+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fcb74084f60 con 0x7fcb7c0a9750 2024-03-29T11:26:14.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.659+0000 7fcb6affd640 10 client.4505.objecter ms_dispatch 0x7fcb7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:14.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.659+0000 7fcb6affd640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:14.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.659+0000 7fcb6affd640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:14.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.659+0000 7fcb6affd640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T11:26:14.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.659+0000 7fcb79d9b640 10 monclient: get_auth_request con 0x7fcb5c06e040 auth_method 0 2024-03-29T11:26:14.649 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.659+0000 7fcb6affd640 10 client.4505.objecter ms_handle_connect 0x7fcb5c06e040 2024-03-29T11:26:14.652 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.666+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb74050860 con 0x7fcb7c0a9750 2024-03-29T11:26:14.652 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.666+0000 7fcb6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:14.652 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:14.666+0000 7fcb6affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:14.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.010+0000 7fcb7b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:14.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.010+0000 7fcb7b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:14.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.010+0000 7fcb7b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:14.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.010+0000 7fcb7b59e640 1 -- 172.21.15.67:0/519326607 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb60005330 con 0x7fcb7c0a9750 2024-03-29T11:26:14.997 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.011+0000 7fcb6affd640 1 -- 172.21.15.67:0/519326607 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fcb74052790 con 0x7fcb7c0a9750 2024-03-29T11:26:14.997 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.011+0000 7fcb6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:14.997 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.011+0000 7fcb6affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:14.997 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:26:14.998 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:15.000 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.014+0000 7fcb7b59e640 10 librados: watch_flush enter 2024-03-29T11:26:15.000 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.015+0000 7fcb7b59e640 10 librados: watch_flush exit 2024-03-29T11:26:15.000 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.015+0000 7fcb7b59e640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T11:26:15.000 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.015+0000 7fcb7b59e640 10 client.4505.objecter successfully canceled tick 2024-03-29T11:26:15.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.015+0000 7fcb7b59e640 1 -- 172.21.15.67:0/519326607 >> v1:172.21.15.67:6816/404787197 conn(0x7fcb5c06e040 legacy=0x7fcb5c070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.015+0000 7fcb7b59e640 10 monclient: shutdown 2024-03-29T11:26:15.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.015+0000 7fcb7b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:15.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.015+0000 7fcb7b59e640 1 -- 172.21.15.67:0/519326607 >> v1:172.21.15.67:6790/0 conn(0x7fcb7c0a9750 legacy=0x7fcb7c1c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.016+0000 7fcb7b59e640 1 -- 172.21.15.67:0/519326607 shutdown_connections 2024-03-29T11:26:15.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.016+0000 7fcb7b59e640 1 -- 172.21.15.67:0/519326607 >> 172.21.15.67:0/519326607 conn(0x7fcb7c0673e0 msgr2=0x7fcb7c0b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:15.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.017+0000 7fcb7b59e640 1 -- 172.21.15.67:0/519326607 shutdown_connections 2024-03-29T11:26:15.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.017+0000 7fcb7b59e640 1 -- 172.21.15.67:0/519326607 wait complete. 2024-03-29T11:26:15.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:15.017+0000 7fcb7b59e640 1 librados: shutdown 2024-03-29T11:26:15.016 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:21.822 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:21.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.987+0000 7fad54744640 10 monclient: get_monmap_and_config 2024-03-29T11:26:21.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.987+0000 7fad54744640 10 monclient: build_initial_monmap 2024-03-29T11:26:21.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.987+0000 7fad54744640 10 monclient: monmap: 2024-03-29T11:26:21.974 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:21.974 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:21.974 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:26:21.989336+0000 2024-03-29T11:26:21.975 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:26:21.989336+0000 2024-03-29T11:26:21.975 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:21.975 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:21.975 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:21.975 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:26:21.975 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:26:21.975 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:21.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.988+0000 7fad54744640 1 Processor -- start 2024-03-29T11:26:21.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.988+0000 7fad54744640 1 -- start start 2024-03-29T11:26:21.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.988+0000 7fad54744640 10 monclient: init 2024-03-29T11:26:21.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.989+0000 7fad54744640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:21.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.989+0000 7fad54744640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:21.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.993+0000 7fad54744640 10 monclient(hunting): picked mon.noname-b con 0x7fad4c1422b0 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:21.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.993+0000 7fad54744640 10 monclient(hunting): picked mon.noname-a con 0x7fad4c142690 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:21.984 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.997+0000 7fad54744640 10 monclient(hunting): picked mon.noname-c con 0x7fad4c0b17d0 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:21.984 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.997+0000 7fad54744640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c142c80 con 0x7fad4c142690 2024-03-29T11:26:21.984 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.997+0000 7fad54744640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c0aae40 con 0x7fad4c0b17d0 2024-03-29T11:26:21.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.997+0000 7fad54744640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c13fbd0 con 0x7fad4c1422b0 2024-03-29T11:26:21.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.997+0000 7fad54744640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:21.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.997+0000 7fad54744640 10 monclient(hunting): authenticate will time out at 1242.294459s 2024-03-29T11:26:21.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.997+0000 7fad524e0640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fad4c1422b0 0x7fad4c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:41259/0 (socket says 172.21.15.67:41259) 2024-03-29T11:26:21.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.997+0000 7fad524e0640 1 -- 172.21.15.67:0/2105255596 learned_addr learned my addr 172.21.15.67:0/2105255596 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:21.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.998+0000 7fad514de640 1 -- 172.21.15.67:0/2105255596 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657267473 0 0) 0x7fad4c13fbd0 con 0x7fad4c1422b0 2024-03-29T11:26:21.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.998+0000 7fad514de640 10 monclient(hunting): _negotiate 2024-03-29T11:26:21.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.998+0000 7fad514de640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:21.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.998+0000 7fad514de640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:21.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.998+0000 7fad514de640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:26:21.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.998+0000 7fad514de640 1 -- 172.21.15.67:0/2105255596 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad38003520 con 0x7fad4c1422b0 2024-03-29T11:26:21.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.998+0000 7fad514de640 1 -- 172.21.15.67:0/2105255596 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1158617419 0 0) 0x7fad38003520 con 0x7fad4c1422b0 2024-03-29T11:26:21.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.998+0000 7fad514de640 1 -- 172.21.15.67:0/2105255596 >> v1:172.21.15.82:6789/0 conn(0x7fad4c0b17d0 legacy=0x7fad4c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:21.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.998+0000 7fad514de640 1 -- 172.21.15.67:0/2105255596 >> v1:172.21.15.67:6789/0 conn(0x7fad4c142690 legacy=0x7fad4c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:21.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.998+0000 7fad514de640 10 monclient: _finish_hunting 0 2024-03-29T11:26:21.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.998+0000 7fad514de640 1 monclient: found mon.noname-b 2024-03-29T11:26:21.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.998+0000 7fad514de640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:21.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:26:21.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 1 -- 172.21.15.67:0/2105255596 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c151940 con 0x7fad4c1422b0 2024-03-29T11:26:21.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 10 monclient: _finish_auth 0 2024-03-29T11:26:21.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 10 monclient: _check_auth_tickets 2024-03-29T11:26:21.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:21.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 1 -- 172.21.15.67:0/2105255596 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fad48001690 con 0x7fad4c1422b0 2024-03-29T11:26:21.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:21.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:21.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 10 monclient: dump: 2024-03-29T11:26:21.987 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:21.988 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:21.988 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:21.988 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:21.988 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:21.988 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:21.988 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:21.988 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:21.988 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:21.989 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:21.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad54744640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:26:21.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad54744640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:21.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 1 -- 172.21.15.67:0/2105255596 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad480017f0 con 0x7fad4c1422b0 2024-03-29T11:26:21.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:21.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 1 -- 172.21.15.67:0/2105255596 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fad48001c40 con 0x7fad4c1422b0 2024-03-29T11:26:21.990 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:21.990 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:21.990 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad514de640 10 monclient: dump: 2024-03-29T11:26:21.990 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:21.990 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:21.990 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:21.990 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:21.990 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:21.990 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:21.991 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:21.991 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:21.991 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:21.991 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:21.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad54744640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:21.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:21.999+0000 7fad54744640 4 set_mon_vals no callback set 2024-03-29T11:26:21.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.000+0000 7fad54744640 10 monclient: shutdown 2024-03-29T11:26:21.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.000+0000 7fad54744640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:21.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.000+0000 7fad54744640 1 -- 172.21.15.67:0/2105255596 >> v1:172.21.15.67:6790/0 conn(0x7fad4c1422b0 legacy=0x7fad4c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:21.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.001+0000 7fad54744640 1 -- 172.21.15.67:0/2105255596 shutdown_connections 2024-03-29T11:26:21.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.001+0000 7fad54744640 1 -- 172.21.15.67:0/2105255596 >> 172.21.15.67:0/2105255596 conn(0x7fad4c066a00 msgr2=0x7fad4c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:21.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.001+0000 7fad54744640 1 -- 172.21.15.67:0/2105255596 shutdown_connections 2024-03-29T11:26:21.994 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.001+0000 7fad54744640 1 -- 172.21.15.67:0/2105255596 wait complete. 2024-03-29T11:26:21.994 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.002+0000 7fad54744640 10 monclient: build_initial_monmap 2024-03-29T11:26:21.994 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.002+0000 7fad54744640 10 monclient: monmap: 2024-03-29T11:26:21.994 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:21.994 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:21.994 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:26:21.994 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:26:21.994 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:21.994 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:21.995 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:21.995 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:26:21.995 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:26:21.995 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:21.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.002+0000 7fad54744640 1 librados: starting msgr at 2024-03-29T11:26:21.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.002+0000 7fad54744640 1 librados: starting objecter 2024-03-29T11:26:21.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.003+0000 7fad54744640 1 Processor -- start 2024-03-29T11:26:21.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.003+0000 7fad54744640 1 -- start start 2024-03-29T11:26:21.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.003+0000 7fad54744640 1 librados: setting wanted keys 2024-03-29T11:26:21.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.003+0000 7fad54744640 1 librados: calling monclient init 2024-03-29T11:26:21.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.003+0000 7fad54744640 10 monclient: init 2024-03-29T11:26:21.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.004+0000 7fad54744640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:21.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.004+0000 7fad54744640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:21.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.005+0000 7fad54744640 10 monclient(hunting): picked mon.noname-b con 0x7fad4c0b17d0 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:21.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.006+0000 7fad54744640 10 monclient(hunting): picked mon.noname-a con 0x7fad4c1422b0 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:21.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.006+0000 7fad54744640 10 monclient(hunting): picked mon.noname-c con 0x7fad4c142690 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:21.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.006+0000 7fad54744640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c131d50 con 0x7fad4c1422b0 2024-03-29T11:26:21.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.006+0000 7fad54744640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c06eab0 con 0x7fad4c0b17d0 2024-03-29T11:26:21.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.006+0000 7fad54744640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c06ed90 con 0x7fad4c142690 2024-03-29T11:26:21.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.006+0000 7fad54744640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:21.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.006+0000 7fad54744640 10 monclient(hunting): authenticate will time out at 1242.303609s 2024-03-29T11:26:21.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.007+0000 7fad524e0640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fad4c0b17d0 0x7fad4c1321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:39431/0 (socket says 172.21.15.67:39431) 2024-03-29T11:26:21.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.007+0000 7fad524e0640 1 -- 172.21.15.67:0/2973931517 learned_addr learned my addr 172.21.15.67:0/2973931517 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:21.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.007+0000 7fad42ffd640 10 client.?.objecter ms_handle_connect 0x7fad4c0b17d0 2024-03-29T11:26:21.996 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.007+0000 7fad42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:21.998 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.008+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108904531 0 0) 0x7fad4c06eab0 con 0x7fad4c0b17d0 2024-03-29T11:26:21.998 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.008+0000 7fad42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:21.998 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.008+0000 7fad42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:21.998 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.008+0000 7fad42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:21.998 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.008+0000 7fad42ffd640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:26:21.999 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.008+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad30004eb0 con 0x7fad4c0b17d0 2024-03-29T11:26:21.999 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.009+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 642326994 0 0) 0x7fad30004eb0 con 0x7fad4c0b17d0 2024-03-29T11:26:21.999 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.009+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad4c06eab0 con 0x7fad4c0b17d0 2024-03-29T11:26:21.999 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.009+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fad48001bc0 con 0x7fad4c0b17d0 2024-03-29T11:26:21.999 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.009+0000 7fad42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:21.999 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.009+0000 7fad42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:21.999 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.009+0000 7fad42ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:21.999 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:21.999 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:21.999 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:21.999 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:21.999 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:21.999 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:21.999 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:22.000 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:22.000 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:22.000 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:22.000 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.009+0000 7fad42ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:22.000 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.010+0000 7fad42ffd640 10 client.?.objecter ms_handle_connect 0x7fad4c142690 2024-03-29T11:26:22.000 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.010+0000 7fad42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:22.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.012+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3619542988 0 0) 0x7fad4c06eab0 con 0x7fad4c0b17d0 2024-03-29T11:26:22.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.012+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 >> v1:172.21.15.67:6790/0 conn(0x7fad4c142690 legacy=0x7fad4c071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.012+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 >> v1:172.21.15.67:6789/0 conn(0x7fad4c1422b0 legacy=0x7fad4c071460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.012+0000 7fad42ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:22.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.012+0000 7fad42ffd640 1 monclient: found mon.b 2024-03-29T11:26:22.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.012+0000 7fad42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:22.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.012+0000 7fad42ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:26:22.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.012+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c1328f0 con 0x7fad4c0b17d0 2024-03-29T11:26:22.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.012+0000 7fad42ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:22.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.012+0000 7fad42ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:22.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.012+0000 7fad42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:22.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad54744640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:26:22.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad54744640 10 monclient: _renew_subs 2024-03-29T11:26:22.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad54744640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:26:22.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad54744640 1 -- 172.21.15.67:0/2973931517 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4c1deae0 con 0x7fad4c0b17d0 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad54744640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad54744640 10 monclient: _renew_subs 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad54744640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad54744640 1 -- 172.21.15.67:0/2973931517 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4c13bc60 con 0x7fad4c0b17d0 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad54744640 1 librados: init done 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad48004bf0 con 0x7fad4c0b17d0 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fad48001d20 con 0x7fad4c0b17d0 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad42ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad42ffd640 10 monclient: dump: 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:22.003 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:22.004 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.004 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:22.004 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:22.004 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:22.004 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:22.004 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:22.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.013+0000 7fad50cdd640 4 set_mon_vals no callback set 2024-03-29T11:26:22.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.016+0000 7fad54744640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.016+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fad48004f00 con 0x7fad4c0b17d0 2024-03-29T11:26:22.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.017+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fad48084d00 con 0x7fad4c0b17d0 2024-03-29T11:26:22.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.017+0000 7fad42ffd640 10 client.4429.objecter ms_dispatch 0x7fad4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:22.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.017+0000 7fad42ffd640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:22.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.017+0000 7fad42ffd640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:22.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.017+0000 7fad42ffd640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-29T11:26:22.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.018+0000 7fad54744640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.018+0000 7fad54744640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:26:22.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.018+0000 7fad54744640 1 -- 172.21.15.67:0/2973931517 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad34005080 con 0x7fad4c0b17d0 2024-03-29T11:26:22.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.018+0000 7fad51cdf640 10 monclient: get_auth_request con 0x7fad3006dcf0 auth_method 0 2024-03-29T11:26:22.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.022+0000 7fad42ffd640 10 client.4429.objecter ms_handle_connect 0x7fad3006dcf0 2024-03-29T11:26:22.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.027+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad480505e0 con 0x7fad4c0b17d0 2024-03-29T11:26:22.015 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.027+0000 7fad42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.015 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.027+0000 7fad42ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:22.345 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.358+0000 7fad54744640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:22.345 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.358+0000 7fad54744640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:22.345 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.358+0000 7fad54744640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:26:22.345 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.358+0000 7fad54744640 1 -- 172.21.15.67:0/2973931517 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad34005330 con 0x7fad4c0b17d0 2024-03-29T11:26:22.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.359+0000 7fad42ffd640 1 -- 172.21.15.67:0/2973931517 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fad48052510 con 0x7fad4c0b17d0 2024-03-29T11:26:22.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.359+0000 7fad42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:22.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.359+0000 7fad42ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:22.349 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:26:22.349 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:22.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.363+0000 7fad54744640 10 librados: watch_flush enter 2024-03-29T11:26:22.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.363+0000 7fad54744640 10 librados: watch_flush exit 2024-03-29T11:26:22.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.363+0000 7fad54744640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-29T11:26:22.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.363+0000 7fad54744640 10 client.4429.objecter successfully canceled tick 2024-03-29T11:26:22.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.363+0000 7fad54744640 1 -- 172.21.15.67:0/2973931517 >> v1:172.21.15.67:6816/404787197 conn(0x7fad3006dcf0 legacy=0x7fad300701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.363+0000 7fad54744640 10 monclient: shutdown 2024-03-29T11:26:22.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.363+0000 7fad54744640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:22.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.363+0000 7fad54744640 1 -- 172.21.15.67:0/2973931517 >> v1:172.21.15.82:6789/0 conn(0x7fad4c0b17d0 legacy=0x7fad4c1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.367+0000 7fad54744640 1 -- 172.21.15.67:0/2973931517 shutdown_connections 2024-03-29T11:26:22.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.367+0000 7fad54744640 1 -- 172.21.15.67:0/2973931517 >> 172.21.15.67:0/2973931517 conn(0x7fad4c066a00 msgr2=0x7fad4c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:22.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.367+0000 7fad54744640 1 -- 172.21.15.67:0/2973931517 shutdown_connections 2024-03-29T11:26:22.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.367+0000 7fad54744640 1 -- 172.21.15.67:0/2973931517 wait complete. 2024-03-29T11:26:22.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:22.368+0000 7fad54744640 1 librados: shutdown 2024-03-29T11:26:22.371 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:29.179 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:29.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.339+0000 7ff93d941640 10 monclient: get_monmap_and_config 2024-03-29T11:26:29.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.339+0000 7ff93d941640 10 monclient: build_initial_monmap 2024-03-29T11:26:29.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.339+0000 7ff93d941640 10 monclient: monmap: 2024-03-29T11:26:29.325 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:29.325 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:29.325 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:26:29.339957+0000 2024-03-29T11:26:29.325 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:26:29.339957+0000 2024-03-29T11:26:29.325 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:29.326 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:29.326 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:29.326 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:26:29.326 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:26:29.326 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:29.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.340+0000 7ff93d941640 1 Processor -- start 2024-03-29T11:26:29.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.340+0000 7ff93d941640 1 -- start start 2024-03-29T11:26:29.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.340+0000 7ff93d941640 10 monclient: init 2024-03-29T11:26:29.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.341+0000 7ff93d941640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:29.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.341+0000 7ff93d941640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:29.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.341+0000 7ff93d941640 10 monclient(hunting): picked mon.noname-c con 0x7ff9380b4820 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:29.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.341+0000 7ff93d941640 10 monclient(hunting): picked mon.noname-b con 0x7ff9380a9ce0 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:29.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.341+0000 7ff93d941640 10 monclient(hunting): picked mon.noname-a con 0x7ff9380b3cd0 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:29.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.341+0000 7ff93d941640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9380b4180 con 0x7ff9380b3cd0 2024-03-29T11:26:29.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.341+0000 7ff93d941640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9380aaf60 con 0x7ff9380b4820 2024-03-29T11:26:29.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.341+0000 7ff93d941640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff93813c200 con 0x7ff9380a9ce0 2024-03-29T11:26:29.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.341+0000 7ff93d941640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:29.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.341+0000 7ff93d941640 10 monclient(hunting): authenticate will time out at 1249.637142s 2024-03-29T11:26:29.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.342+0000 7ff937fff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7ff9380a9ce0 0x7ff938145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:42762/0 (socket says 172.21.15.67:42762) 2024-03-29T11:26:29.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.342+0000 7ff937fff640 1 -- 172.21.15.67:0/599918559 learned_addr learned my addr 172.21.15.67:0/599918559 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:29.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.343+0000 7ff9377fe640 1 -- 172.21.15.67:0/599918559 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403262087 0 0) 0x7ff93813c200 con 0x7ff9380a9ce0 2024-03-29T11:26:29.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.343+0000 7ff9377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:29.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.343+0000 7ff9377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:29.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.343+0000 7ff9377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:29.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.343+0000 7ff9377fe640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:26:29.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.343+0000 7ff9377fe640 1 -- 172.21.15.67:0/599918559 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff924003520 con 0x7ff9380a9ce0 2024-03-29T11:26:29.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.343+0000 7ff9377fe640 1 -- 172.21.15.67:0/599918559 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2869812038 0 0) 0x7ff924003520 con 0x7ff9380a9ce0 2024-03-29T11:26:29.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.343+0000 7ff9377fe640 1 -- 172.21.15.67:0/599918559 >> v1:172.21.15.82:6789/0 conn(0x7ff9380b4820 legacy=0x7ff93813baf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:26:29.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.343+0000 7ff9377fe640 1 -- 172.21.15.67:0/599918559 >> v1:172.21.15.67:6789/0 conn(0x7ff9380b3cd0 legacy=0x7ff93814d7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:29.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff9377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:29.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff9377fe640 1 monclient: found mon.noname-b 2024-03-29T11:26:29.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff9377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:29.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff9377fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:26:29.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff9377fe640 1 -- 172.21.15.67:0/599918559 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff938145e80 con 0x7ff9380a9ce0 2024-03-29T11:26:29.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff9377fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:29.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff9377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:29.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff9377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:29.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff93d941640 5 monclient: authenticate success, global_id 4514 2024-03-29T11:26:29.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff93d941640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:29.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff9377fe640 1 -- 172.21.15.67:0/599918559 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff920001690 con 0x7ff9380a9ce0 2024-03-29T11:26:29.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff9377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:29.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff9377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:29.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff9377fe640 10 monclient: dump: 2024-03-29T11:26:29.334 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:29.334 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:29.334 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:29.334 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:29.334 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:29.334 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:29.334 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:29.334 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:29.334 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:29.334 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:29.334 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.347+0000 7ff93d941640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:29.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.348+0000 7ff9377fe640 1 -- 172.21.15.67:0/599918559 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff920001920 con 0x7ff9380a9ce0 2024-03-29T11:26:29.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.348+0000 7ff9377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:29.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.348+0000 7ff93d941640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:29.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.348+0000 7ff93d941640 4 set_mon_vals no callback set 2024-03-29T11:26:29.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.348+0000 7ff9377fe640 1 -- 172.21.15.67:0/599918559 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff920001cb0 con 0x7ff9380a9ce0 2024-03-29T11:26:29.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.348+0000 7ff9377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:29.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.348+0000 7ff9377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:29.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.348+0000 7ff9377fe640 10 monclient: dump: 2024-03-29T11:26:29.335 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:29.335 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:29.336 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:29.336 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:29.336 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:29.336 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:29.336 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:29.336 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:29.336 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:29.336 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:29.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.349+0000 7ff93d941640 10 monclient: shutdown 2024-03-29T11:26:29.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.349+0000 7ff93d941640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:29.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.349+0000 7ff93d941640 1 -- 172.21.15.67:0/599918559 >> v1:172.21.15.67:6790/0 conn(0x7ff9380a9ce0 legacy=0x7ff938145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:29.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.349+0000 7ff93d941640 1 -- 172.21.15.67:0/599918559 shutdown_connections 2024-03-29T11:26:29.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.349+0000 7ff93d941640 1 -- 172.21.15.67:0/599918559 >> 172.21.15.67:0/599918559 conn(0x7ff93806d070 msgr2=0x7ff938066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:29.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.349+0000 7ff93d941640 1 -- 172.21.15.67:0/599918559 shutdown_connections 2024-03-29T11:26:29.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.349+0000 7ff93d941640 1 -- 172.21.15.67:0/599918559 wait complete. 2024-03-29T11:26:29.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.350+0000 7ff93d941640 10 monclient: build_initial_monmap 2024-03-29T11:26:29.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.350+0000 7ff93d941640 10 monclient: monmap: 2024-03-29T11:26:29.338 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:29.338 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:29.338 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:26:29.338 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:26:29.338 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:29.338 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:29.338 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:29.338 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:26:29.338 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:26:29.338 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:29.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.351+0000 7ff93d941640 1 librados: starting msgr at 2024-03-29T11:26:29.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.351+0000 7ff93d941640 1 librados: starting objecter 2024-03-29T11:26:29.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.351+0000 7ff93d941640 1 Processor -- start 2024-03-29T11:26:29.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.351+0000 7ff93d941640 1 -- start start 2024-03-29T11:26:29.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.351+0000 7ff93d941640 1 librados: setting wanted keys 2024-03-29T11:26:29.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.351+0000 7ff93d941640 1 librados: calling monclient init 2024-03-29T11:26:29.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.351+0000 7ff93d941640 10 monclient: init 2024-03-29T11:26:29.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.352+0000 7ff93d941640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:29.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.352+0000 7ff93d941640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:29.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.352+0000 7ff93d941640 10 monclient(hunting): picked mon.noname-a con 0x7ff9380a9ce0 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:29.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.355+0000 7ff93d941640 10 monclient(hunting): picked mon.noname-c con 0x7ff9380b3cd0 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:29.344 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.355+0000 7ff93d941640 10 monclient(hunting): picked mon.noname-b con 0x7ff9380b4820 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:29.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.358+0000 7ff93d941640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff93813b9a0 con 0x7ff9380a9ce0 2024-03-29T11:26:29.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.358+0000 7ff93d941640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9381da5c0 con 0x7ff9380b4820 2024-03-29T11:26:29.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.358+0000 7ff93d941640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9381db1f0 con 0x7ff9380b3cd0 2024-03-29T11:26:29.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.358+0000 7ff93d941640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:29.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.358+0000 7ff93d941640 10 monclient(hunting): authenticate will time out at 1249.654208s 2024-03-29T11:26:29.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.356+0000 7ff937fff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7ff9380b3cd0 0x7ff938132d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:47806/0 (socket says 172.21.15.67:47806) 2024-03-29T11:26:29.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.358+0000 7ff937fff640 1 -- 172.21.15.67:0/3811220399 learned_addr learned my addr 172.21.15.67:0/3811220399 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:29.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.359+0000 7ff9357fa640 10 client.?.objecter ms_handle_connect 0x7ff9380b3cd0 2024-03-29T11:26:29.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.359+0000 7ff9357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:29.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.359+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657222555 0 0) 0x7ff9381db1f0 con 0x7ff9380b3cd0 2024-03-29T11:26:29.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.359+0000 7ff9357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:29.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.359+0000 7ff9357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:29.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.359+0000 7ff9357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:29.347 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.359+0000 7ff9357fa640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:26:29.347 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.359+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff918004eb0 con 0x7ff9380b3cd0 2024-03-29T11:26:29.347 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.358+0000 7ff93c93f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff9380a9ce0 0x7ff9381daae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58017/0 (socket says 172.21.15.67:58017) 2024-03-29T11:26:29.347 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.359+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 132926158 0 0) 0x7ff918004eb0 con 0x7ff9380b3cd0 2024-03-29T11:26:29.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.359+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9381db1f0 con 0x7ff9380b3cd0 2024-03-29T11:26:29.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.360+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff920001610 con 0x7ff9380b3cd0 2024-03-29T11:26:29.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.360+0000 7ff9357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:29.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.360+0000 7ff9357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:29.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.360+0000 7ff9357fa640 10 monclient(hunting): dump: 2024-03-29T11:26:29.348 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:29.348 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.360+0000 7ff9357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 10 client.?.objecter ms_handle_connect 0x7ff9380b4820 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 10 client.?.objecter ms_handle_connect 0x7ff9380a9ce0 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3444659796 0 0) 0x7ff9381db1f0 con 0x7ff9380b3cd0 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 >> v1:172.21.15.82:6789/0 conn(0x7ff9380b4820 legacy=0x7ff9381e27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:29.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 >> v1:172.21.15.67:6789/0 conn(0x7ff9380a9ce0 legacy=0x7ff9381daae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:29.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 10 monclient: _finish_hunting 0 2024-03-29T11:26:29.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 1 monclient: found mon.c 2024-03-29T11:26:29.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:29.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:29.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9381db380 con 0x7ff9380b3cd0 2024-03-29T11:26:29.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 10 monclient: _finish_auth 0 2024-03-29T11:26:29.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 10 monclient: _check_auth_tickets 2024-03-29T11:26:29.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff9357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:29.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff93d941640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:26:29.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff93d941640 10 monclient: _renew_subs 2024-03-29T11:26:29.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff93d941640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:29.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff93d941640 1 -- 172.21.15.67:0/3811220399 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff938134420 con 0x7ff9380b3cd0 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff93d941640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff93d941640 10 monclient: _renew_subs 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff93d941640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff93d941640 1 -- 172.21.15.67:0/3811220399 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff938144380 con 0x7ff9380b3cd0 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.361+0000 7ff93d941640 1 librados: init done 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.362+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff920004bf0 con 0x7ff9380b3cd0 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.362+0000 7ff9357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.362+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff920001770 con 0x7ff9380b3cd0 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.362+0000 7ff9357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.362+0000 7ff9357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.362+0000 7ff9357fa640 10 monclient: dump: 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:29.352 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:29.352 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:29.352 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:29.352 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:29.352 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:29.352 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:29.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.362+0000 7ff936ffd640 4 set_mon_vals no callback set 2024-03-29T11:26:29.353 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.363+0000 7ff93d941640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:29.353 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.363+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7ff92001ad70 con 0x7ff9380b3cd0 2024-03-29T11:26:29.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.364+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7ff920084cc0 con 0x7ff9380b3cd0 2024-03-29T11:26:29.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.364+0000 7ff9357fa640 10 client.4517.objecter ms_dispatch 0x7ff938000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:29.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.364+0000 7ff9357fa640 3 client.4517.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:29.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.364+0000 7ff9357fa640 3 client.4517.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:29.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.364+0000 7ff9357fa640 20 client.4517.objecter dump_active .. 0 homeless 2024-03-29T11:26:29.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.364+0000 7ff93d941640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:29.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.364+0000 7ff93d941640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:29.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.364+0000 7ff93d941640 1 -- 172.21.15.67:0/3811220399 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff91c005080 con 0x7ff9380b3cd0 2024-03-29T11:26:29.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.365+0000 7ff93c93f640 10 monclient: get_auth_request con 0x7ff91806dc50 auth_method 0 2024-03-29T11:26:29.354 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.365+0000 7ff9357fa640 10 client.4517.objecter ms_handle_connect 0x7ff91806dc50 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.373+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff9200503d0 con 0x7ff9380b3cd0 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.373+0000 7ff9357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.373+0000 7ff9357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:29.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.711+0000 7ff93d941640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:29.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.711+0000 7ff93d941640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:29.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.711+0000 7ff93d941640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:29.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.711+0000 7ff93d941640 1 -- 172.21.15.67:0/3811220399 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff91c005330 con 0x7ff9380b3cd0 2024-03-29T11:26:29.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.713+0000 7ff9357fa640 1 -- 172.21.15.67:0/3811220399 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7ff9200524d0 con 0x7ff9380b3cd0 2024-03-29T11:26:29.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.713+0000 7ff9357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:29.701 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.713+0000 7ff9357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:29.701 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:26:29.701 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:29.704 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.718+0000 7ff93d941640 10 librados: watch_flush enter 2024-03-29T11:26:29.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.719+0000 7ff93d941640 10 librados: watch_flush exit 2024-03-29T11:26:29.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.719+0000 7ff93d941640 20 client.4517.objecter shutdown clearing up homeless session... 2024-03-29T11:26:29.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.719+0000 7ff93d941640 10 client.4517.objecter successfully canceled tick 2024-03-29T11:26:29.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.719+0000 7ff93d941640 1 -- 172.21.15.67:0/3811220399 >> v1:172.21.15.67:6816/404787197 conn(0x7ff91806dc50 legacy=0x7ff918070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:29.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.719+0000 7ff93d941640 10 monclient: shutdown 2024-03-29T11:26:29.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.719+0000 7ff93d941640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:29.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.719+0000 7ff93d941640 1 -- 172.21.15.67:0/3811220399 >> v1:172.21.15.67:6790/0 conn(0x7ff9380b3cd0 legacy=0x7ff938132d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:29.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.720+0000 7ff93d941640 1 -- 172.21.15.67:0/3811220399 shutdown_connections 2024-03-29T11:26:29.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.720+0000 7ff93d941640 1 -- 172.21.15.67:0/3811220399 >> 172.21.15.67:0/3811220399 conn(0x7ff93806d070 msgr2=0x7ff9380b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:29.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.721+0000 7ff93d941640 1 -- 172.21.15.67:0/3811220399 shutdown_connections 2024-03-29T11:26:29.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.721+0000 7ff93d941640 1 -- 172.21.15.67:0/3811220399 wait complete. 2024-03-29T11:26:29.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:29.721+0000 7ff93d941640 1 librados: shutdown 2024-03-29T11:26:29.724 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:36.532 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:36.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.693+0000 7fa4e0bfd640 10 monclient: get_monmap_and_config 2024-03-29T11:26:36.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.693+0000 7fa4e0bfd640 10 monclient: build_initial_monmap 2024-03-29T11:26:36.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.693+0000 7fa4e0bfd640 10 monclient: monmap: 2024-03-29T11:26:36.680 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:36.680 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:36.680 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:26:36.693888+0000 2024-03-29T11:26:36.681 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:26:36.693888+0000 2024-03-29T11:26:36.681 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:36.681 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:36.681 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:36.681 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:26:36.681 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:26:36.682 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:36.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.694+0000 7fa4e0bfd640 1 Processor -- start 2024-03-29T11:26:36.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.694+0000 7fa4e0bfd640 1 -- start start 2024-03-29T11:26:36.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.694+0000 7fa4e0bfd640 10 monclient: init 2024-03-29T11:26:36.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.695+0000 7fa4e0bfd640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:36.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.695+0000 7fa4e0bfd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:36.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.697+0000 7fa4e0bfd640 10 monclient(hunting): picked mon.noname-c con 0x7fa4d4006a70 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:36.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.697+0000 7fa4e0bfd640 10 monclient(hunting): picked mon.noname-a con 0x7fa4d402b100 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:36.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.697+0000 7fa4e0bfd640 10 monclient(hunting): picked mon.noname-b con 0x7fa4d4029c30 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:36.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.697+0000 7fa4e0bfd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d402a270 con 0x7fa4d402b100 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.697+0000 7fa4e0bfd640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d402c440 con 0x7fa4d4006a70 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.697+0000 7fa4e0bfd640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d40c9440 con 0x7fa4d4029c30 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.698+0000 7fa4e0bfd640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.698+0000 7fa4e0bfd640 10 monclient(hunting): authenticate will time out at 1256.993991s 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.698+0000 7fa4dbfff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fa4d4029c30 0x7fa4d40d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:42692/0 (socket says 172.21.15.67:42692) 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.698+0000 7fa4dbfff640 1 -- 172.21.15.67:0/3209612295 learned_addr learned my addr 172.21.15.67:0/3209612295 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.699+0000 7fa4da7fc640 1 -- 172.21.15.67:0/3209612295 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095518675 0 0) 0x7fa4d40c9440 con 0x7fa4d4029c30 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.699+0000 7fa4da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.699+0000 7fa4da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.699+0000 7fa4da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:36.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.699+0000 7fa4da7fc640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:26:36.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.699+0000 7fa4da7fc640 1 -- 172.21.15.67:0/3209612295 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4cc003520 con 0x7fa4d4029c30 2024-03-29T11:26:36.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.700+0000 7fa4da7fc640 1 -- 172.21.15.67:0/3209612295 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2468662079 0 0) 0x7fa4cc003520 con 0x7fa4d4029c30 2024-03-29T11:26:36.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.700+0000 7fa4da7fc640 1 -- 172.21.15.67:0/3209612295 >> v1:172.21.15.82:6789/0 conn(0x7fa4d4006a70 legacy=0x7fa4d40c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:36.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.700+0000 7fa4da7fc640 1 -- 172.21.15.67:0/3209612295 >> v1:172.21.15.67:6789/0 conn(0x7fa4d402b100 legacy=0x7fa4d40cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:36.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.700+0000 7fa4da7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:36.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.700+0000 7fa4da7fc640 1 monclient: found mon.noname-b 2024-03-29T11:26:36.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.700+0000 7fa4da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:36.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.700+0000 7fa4da7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:26:36.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.700+0000 7fa4da7fc640 1 -- 172.21.15.67:0/3209612295 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d40c95d0 con 0x7fa4d4029c30 2024-03-29T11:26:36.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.701+0000 7fa4da7fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:36.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.701+0000 7fa4da7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:36.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.701+0000 7fa4da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:36.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.701+0000 7fa4da7fc640 1 -- 172.21.15.67:0/3209612295 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa4dc04f730 con 0x7fa4d4029c30 2024-03-29T11:26:36.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.701+0000 7fa4da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:36.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.701+0000 7fa4da7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:36.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.701+0000 7fa4da7fc640 10 monclient: dump: 2024-03-29T11:26:36.690 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.701+0000 7fa4e0bfd640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.701+0000 7fa4e0bfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.701+0000 7fa4da7fc640 1 -- 172.21.15.67:0/3209612295 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4dc075f10 con 0x7fa4d4029c30 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.701+0000 7fa4da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.701+0000 7fa4da7fc640 1 -- 172.21.15.67:0/3209612295 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa4dc0758b0 con 0x7fa4d4029c30 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.701+0000 7fa4e0bfd640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.701+0000 7fa4e0bfd640 4 set_mon_vals no callback set 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.702+0000 7fa4e0bfd640 10 monclient: shutdown 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.702+0000 7fa4da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.702+0000 7fa4da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.702+0000 7fa4da7fc640 10 monclient: dump: 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.702+0000 7fa4e0bfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.702+0000 7fa4e0bfd640 1 -- 172.21.15.67:0/3209612295 >> v1:172.21.15.67:6790/0 conn(0x7fa4d4029c30 legacy=0x7fa4d40d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.702+0000 7fa4e0bfd640 1 -- 172.21.15.67:0/3209612295 shutdown_connections 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.702+0000 7fa4e0bfd640 1 -- 172.21.15.67:0/3209612295 >> 172.21.15.67:0/3209612295 conn(0x7fa4d40058f0 msgr2=0x7fa4d4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.703+0000 7fa4e0bfd640 1 -- 172.21.15.67:0/3209612295 shutdown_connections 2024-03-29T11:26:36.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.704+0000 7fa4e0bfd640 1 -- 172.21.15.67:0/3209612295 wait complete. 2024-03-29T11:26:36.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.705+0000 7fa4e0bfd640 10 monclient: build_initial_monmap 2024-03-29T11:26:36.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.705+0000 7fa4e0bfd640 10 monclient: monmap: 2024-03-29T11:26:36.695 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:36.695 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:36.695 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:26:36.695 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:26:36.695 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:36.695 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:36.695 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:36.695 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:26:36.695 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:26:36.695 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:36.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.705+0000 7fa4e0bfd640 1 librados: starting msgr at 2024-03-29T11:26:36.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.705+0000 7fa4e0bfd640 1 librados: starting objecter 2024-03-29T11:26:36.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.706+0000 7fa4e0bfd640 1 Processor -- start 2024-03-29T11:26:36.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.706+0000 7fa4e0bfd640 1 -- start start 2024-03-29T11:26:36.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.706+0000 7fa4e0bfd640 1 librados: setting wanted keys 2024-03-29T11:26:36.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.706+0000 7fa4e0bfd640 1 librados: calling monclient init 2024-03-29T11:26:36.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.706+0000 7fa4e0bfd640 10 monclient: init 2024-03-29T11:26:36.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.707+0000 7fa4e0bfd640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:36.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.707+0000 7fa4e0bfd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:36.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.707+0000 7fa4e0bfd640 10 monclient(hunting): picked mon.noname-c con 0x7fa4d4006a70 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:36.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.707+0000 7fa4e0bfd640 10 monclient(hunting): picked mon.noname-b con 0x7fa4d4029c30 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:36.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.708+0000 7fa4db7fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fa4d4006a70 0x7fa4d40cd9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:43764/0 (socket says 172.21.15.67:43764) 2024-03-29T11:26:36.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.710+0000 7fa4e0bfd640 10 monclient(hunting): picked mon.noname-a con 0x7fa4d40cb070 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:36.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.710+0000 7fa4e0bfd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d40cc2d0 con 0x7fa4d40cb070 2024-03-29T11:26:36.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.710+0000 7fa4e0bfd640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d40cc5b0 con 0x7fa4d4029c30 2024-03-29T11:26:36.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.710+0000 7fa4e0bfd640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d40cc890 con 0x7fa4d4006a70 2024-03-29T11:26:36.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.710+0000 7fa4e0bfd640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:36.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.710+0000 7fa4e0bfd640 10 monclient(hunting): authenticate will time out at 1257.006709s 2024-03-29T11:26:36.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.711+0000 7fa4db7fe640 1 -- 172.21.15.67:0/3712058239 learned_addr learned my addr 172.21.15.67:0/3712058239 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:36.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.711+0000 7fa4c7fff640 10 client.?.objecter ms_handle_connect 0x7fa4d4006a70 2024-03-29T11:26:36.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.711+0000 7fa4c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:36.705 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 client.?.objecter ms_handle_connect 0x7fa4d40cb070 2024-03-29T11:26:36.705 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:36.705 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785538005 0 0) 0x7fa4d40cc2d0 con 0x7fa4d40cb070 2024-03-29T11:26:36.705 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:36.705 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:36.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:36.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:26:36.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c0004eb0 con 0x7fa4d40cb070 2024-03-29T11:26:36.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827790267 0 0) 0x7fa4d40cc890 con 0x7fa4d4006a70 2024-03-29T11:26:36.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:36.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:36.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:36.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:26:36.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d40cc2d0 con 0x7fa4d4006a70 2024-03-29T11:26:36.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 client.?.objecter ms_handle_connect 0x7fa4d4029c30 2024-03-29T11:26:36.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:36.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2849229575 0 0) 0x7fa4d40cc2d0 con 0x7fa4d4006a70 2024-03-29T11:26:36.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4d40cc890 con 0x7fa4d4006a70 2024-03-29T11:26:36.707 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa4d0004000 con 0x7fa4d4006a70 2024-03-29T11:26:36.707 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:36.707 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:36.707 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 monclient(hunting): dump: 2024-03-29T11:26:36.707 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:36.707 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:36.707 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:36.707 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:36.707 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:36.707 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:36.708 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:36.708 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:36.708 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:36.708 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:36.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.712+0000 7fa4c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:36.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2435811869 0 0) 0x7fa4d40cc890 con 0x7fa4d4006a70 2024-03-29T11:26:36.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 >> v1:172.21.15.82:6789/0 conn(0x7fa4d4029c30 legacy=0x7fa4d40bfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:36.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 >> v1:172.21.15.67:6789/0 conn(0x7fa4d40cb070 legacy=0x7fa4d416c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:36.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4c7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:36.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4c7fff640 1 monclient: found mon.c 2024-03-29T11:26:36.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:36.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4c7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:36.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d40cb550 con 0x7fa4d4006a70 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4c7fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4c7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4e0bfd640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4e0bfd640 10 monclient: _renew_subs 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4e0bfd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4e0bfd640 1 -- 172.21.15.67:0/3712058239 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d40ca710 con 0x7fa4d4006a70 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4e0bfd640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4e0bfd640 10 monclient: _renew_subs 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4e0bfd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4e0bfd640 1 -- 172.21.15.67:0/3712058239 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d40ce100 con 0x7fa4d4006a70 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.713+0000 7fa4e0bfd640 1 librados: init done 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.714+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4d0000ed0 con 0x7fa4d4006a70 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.714+0000 7fa4c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:36.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.714+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa4d0004160 con 0x7fa4d4006a70 2024-03-29T11:26:36.710 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.714+0000 7fa4c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:36.710 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.714+0000 7fa4c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:36.710 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.714+0000 7fa4c7fff640 10 monclient: dump: 2024-03-29T11:26:36.710 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:36.710 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:36.710 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:36.710 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:36.710 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:36.710 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:36.710 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:36.710 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:36.710 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:36.710 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:36.710 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.714+0000 7fa4d97fa640 4 set_mon_vals no callback set 2024-03-29T11:26:36.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.715+0000 7fa4c5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:36.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.716+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fa4d00012a0 con 0x7fa4d4006a70 2024-03-29T11:26:36.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.717+0000 7fa4c5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:36.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.717+0000 7fa4c5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:36.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.717+0000 7fa4c5ffb640 1 -- 172.21.15.67:0/3712058239 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4bc005080 con 0x7fa4d4006a70 2024-03-29T11:26:36.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.717+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fa4d0084a50 con 0x7fa4d4006a70 2024-03-29T11:26:36.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.717+0000 7fa4c7fff640 10 client.4523.objecter ms_dispatch 0x7fa4dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:36.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.717+0000 7fa4c7fff640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:36.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.717+0000 7fa4c7fff640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:36.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.717+0000 7fa4c7fff640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-29T11:26:36.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.717+0000 7fa4daffd640 10 monclient: get_auth_request con 0x7fa4c006dcd0 auth_method 0 2024-03-29T11:26:36.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.718+0000 7fa4c7fff640 10 client.4523.objecter ms_handle_connect 0x7fa4c006dcd0 2024-03-29T11:26:36.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.726+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4d0050330 con 0x7fa4d4006a70 2024-03-29T11:26:36.713 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.726+0000 7fa4c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:36.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:36.726+0000 7fa4c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:37.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.058+0000 7fa4c5ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.058+0000 7fa4c5ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.058+0000 7fa4c5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:26:37.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.058+0000 7fa4c5ffb640 1 -- 172.21.15.67:0/3712058239 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4bc005330 con 0x7fa4d4006a70 2024-03-29T11:26:37.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.059+0000 7fa4c7fff640 1 -- 172.21.15.67:0/3712058239 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fa4d0052260 con 0x7fa4d4006a70 2024-03-29T11:26:37.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.059+0000 7fa4c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.059+0000 7fa4c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:37.046 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:26:37.046 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:37.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.063+0000 7fa4c5ffb640 10 librados: watch_flush enter 2024-03-29T11:26:37.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.063+0000 7fa4c5ffb640 10 librados: watch_flush exit 2024-03-29T11:26:37.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.063+0000 7fa4c5ffb640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-29T11:26:37.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.063+0000 7fa4c5ffb640 10 client.4523.objecter successfully canceled tick 2024-03-29T11:26:37.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.063+0000 7fa4c5ffb640 1 -- 172.21.15.67:0/3712058239 >> v1:172.21.15.67:6816/404787197 conn(0x7fa4c006dcd0 legacy=0x7fa4c0070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.063+0000 7fa4c5ffb640 10 monclient: shutdown 2024-03-29T11:26:37.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.063+0000 7fa4c5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:37.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.063+0000 7fa4c5ffb640 1 -- 172.21.15.67:0/3712058239 >> v1:172.21.15.67:6790/0 conn(0x7fa4d4006a70 legacy=0x7fa4d40cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.063+0000 7fa4c5ffb640 1 -- 172.21.15.67:0/3712058239 shutdown_connections 2024-03-29T11:26:37.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.063+0000 7fa4c5ffb640 1 -- 172.21.15.67:0/3712058239 >> 172.21.15.67:0/3712058239 conn(0x7fa4d40058f0 msgr2=0x7fa4d4032d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:37.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.063+0000 7fa4c5ffb640 1 -- 172.21.15.67:0/3712058239 shutdown_connections 2024-03-29T11:26:37.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.063+0000 7fa4c5ffb640 1 -- 172.21.15.67:0/3712058239 wait complete. 2024-03-29T11:26:37.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:37.064+0000 7fa4c5ffb640 1 librados: shutdown 2024-03-29T11:26:37.067 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:43.874 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:44.024 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.037+0000 7fd022ab3640 10 monclient: get_monmap_and_config 2024-03-29T11:26:44.024 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.037+0000 7fd022ab3640 10 monclient: build_initial_monmap 2024-03-29T11:26:44.024 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.038+0000 7fd022ab3640 10 monclient: monmap: 2024-03-29T11:26:44.024 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:44.024 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:44.025 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:26:44.038795+0000 2024-03-29T11:26:44.025 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:26:44.038795+0000 2024-03-29T11:26:44.025 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:44.025 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:44.025 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:44.025 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:26:44.025 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:26:44.025 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:44.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.039+0000 7fd022ab3640 1 Processor -- start 2024-03-29T11:26:44.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.039+0000 7fd022ab3640 1 -- start start 2024-03-29T11:26:44.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.039+0000 7fd022ab3640 10 monclient: init 2024-03-29T11:26:44.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.040+0000 7fd022ab3640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:44.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.040+0000 7fd022ab3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:44.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.040+0000 7fd022ab3640 10 monclient(hunting): picked mon.noname-b con 0x7fd01c071de0 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:44.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.040+0000 7fd022ab3640 10 monclient(hunting): picked mon.noname-c con 0x7fd01c0721c0 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:44.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.040+0000 7fd022ab3640 10 monclient(hunting): picked mon.noname-a con 0x7fd01c145ae0 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:44.026 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.040+0000 7fd022ab3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01c145650 con 0x7fd01c145ae0 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.040+0000 7fd022ab3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01c0ae260 con 0x7fd01c0721c0 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.040+0000 7fd022ab3640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01c1420d0 con 0x7fd01c071de0 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.040+0000 7fd022ab3640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.040+0000 7fd022ab3640 10 monclient(hunting): authenticate will time out at 1264.336052s 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.040+0000 7fd021050640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd01c145ae0 0x7fd01c145f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52494/0 (socket says 172.21.15.67:52494) 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.040+0000 7fd021050640 1 -- 172.21.15.67:0/3825191506 learned_addr learned my addr 172.21.15.67:0/3825191506 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:44.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.041+0000 7fd01b7fe640 1 -- 172.21.15.67:0/3825191506 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371415206 0 0) 0x7fd01c145650 con 0x7fd01c145ae0 2024-03-29T11:26:44.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.041+0000 7fd01b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.041+0000 7fd01b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.041+0000 7fd01b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.041+0000 7fd01b7fe640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:26:44.028 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.041+0000 7fd01b7fe640 1 -- 172.21.15.67:0/3825191506 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd004003520 con 0x7fd01c145ae0 2024-03-29T11:26:44.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.041+0000 7fd01b7fe640 1 -- 172.21.15.67:0/3825191506 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660664374 0 0) 0x7fd01c1420d0 con 0x7fd01c071de0 2024-03-29T11:26:44.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.041+0000 7fd01b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.041+0000 7fd01b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.041+0000 7fd01b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.041+0000 7fd01b7fe640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:26:44.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.041+0000 7fd01b7fe640 1 -- 172.21.15.67:0/3825191506 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd01c145650 con 0x7fd01c071de0 2024-03-29T11:26:44.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 1 -- 172.21.15.67:0/3825191506 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982093343 0 0) 0x7fd01c0ae260 con 0x7fd01c0721c0 2024-03-29T11:26:44.029 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.030 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.030 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.030 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:26:44.030 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 1 -- 172.21.15.67:0/3825191506 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd01c1420d0 con 0x7fd01c0721c0 2024-03-29T11:26:44.030 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 1 -- 172.21.15.67:0/3825191506 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1321257128 0 0) 0x7fd01c145650 con 0x7fd01c071de0 2024-03-29T11:26:44.030 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 1 -- 172.21.15.67:0/3825191506 >> v1:172.21.15.82:6789/0 conn(0x7fd01c0721c0 legacy=0x7fd01c13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.030 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 1 -- 172.21.15.67:0/3825191506 >> v1:172.21.15.67:6789/0 conn(0x7fd01c145ae0 legacy=0x7fd01c145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.030 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:44.030 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:26:44.030 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:44.030 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:26:44.030 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 1 -- 172.21.15.67:0/3825191506 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c0b0590 con 0x7fd01c071de0 2024-03-29T11:26:44.030 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:44.031 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:44.031 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:44.031 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd022ab3640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:26:44.031 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd022ab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:44.031 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 1 -- 172.21.15.67:0/3825191506 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd010001670 con 0x7fd01c071de0 2024-03-29T11:26:44.031 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.031 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:44.031 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient: dump: 2024-03-29T11:26:44.031 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:44.031 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:44.031 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:44.031 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:44.031 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.031 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd022ab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 1 -- 172.21.15.67:0/3825191506 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd010001890 con 0x7fd01c071de0 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 1 -- 172.21.15.67:0/3825191506 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd010001ba0 con 0x7fd01c071de0 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd01b7fe640 10 monclient: dump: 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:44.032 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:44.033 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.033 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:44.033 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:44.033 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:44.033 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:44.033 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:44.033 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd022ab3640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:44.033 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.042+0000 7fd022ab3640 4 set_mon_vals no callback set 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.043+0000 7fd022ab3640 10 monclient: shutdown 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.043+0000 7fd022ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.043+0000 7fd022ab3640 1 -- 172.21.15.67:0/3825191506 >> v1:172.21.15.67:6790/0 conn(0x7fd01c071de0 legacy=0x7fd01c0aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.043+0000 7fd022ab3640 1 -- 172.21.15.67:0/3825191506 shutdown_connections 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.043+0000 7fd022ab3640 1 -- 172.21.15.67:0/3825191506 >> 172.21.15.67:0/3825191506 conn(0x7fd01c06b660 msgr2=0x7fd01c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.043+0000 7fd022ab3640 1 -- 172.21.15.67:0/3825191506 shutdown_connections 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.043+0000 7fd022ab3640 1 -- 172.21.15.67:0/3825191506 wait complete. 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.044+0000 7fd022ab3640 10 monclient: build_initial_monmap 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.044+0000 7fd022ab3640 10 monclient: monmap: 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:26:44.039 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:44.039 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:44.039 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:44.039 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:26:44.039 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:26:44.040 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:44.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.045+0000 7fd022ab3640 1 librados: starting msgr at 2024-03-29T11:26:44.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.045+0000 7fd022ab3640 1 librados: starting objecter 2024-03-29T11:26:44.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.045+0000 7fd022ab3640 1 Processor -- start 2024-03-29T11:26:44.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.045+0000 7fd022ab3640 1 -- start start 2024-03-29T11:26:44.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.045+0000 7fd022ab3640 1 librados: setting wanted keys 2024-03-29T11:26:44.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.045+0000 7fd022ab3640 1 librados: calling monclient init 2024-03-29T11:26:44.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.045+0000 7fd022ab3640 10 monclient: init 2024-03-29T11:26:44.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.046+0000 7fd022ab3640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:44.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.046+0000 7fd022ab3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:44.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.046+0000 7fd022ab3640 10 monclient(hunting): picked mon.noname-c con 0x7fd01c145ae0 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:44.040 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.046+0000 7fd022ab3640 10 monclient(hunting): picked mon.noname-b con 0x7fd01c070ab0 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:44.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.046+0000 7fd02084f640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fd01c145ae0 0x7fd01c0ab310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:60344/0 (socket says 172.21.15.67:60344) 2024-03-29T11:26:44.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.046+0000 7fd02084f640 1 -- 172.21.15.67:0/2752151727 learned_addr learned my addr 172.21.15.67:0/2752151727 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:44.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.046+0000 7fd022ab3640 10 monclient(hunting): picked mon.noname-a con 0x7fd01c0716d0 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:44.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.046+0000 7fd022ab3640 1 -- 172.21.15.67:0/2752151727 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01c070f40 con 0x7fd01c0716d0 2024-03-29T11:26:44.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.046+0000 7fd0197fa640 10 client.?.objecter ms_handle_connect 0x7fd01c0716d0 2024-03-29T11:26:44.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.046+0000 7fd0197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:44.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.046+0000 7fd0197fa640 10 client.?.objecter ms_handle_connect 0x7fd01c145ae0 2024-03-29T11:26:44.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.046+0000 7fd0197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:44.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.046+0000 7fd022ab3640 1 -- 172.21.15.67:0/2752151727 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01c135620 con 0x7fd01c070ab0 2024-03-29T11:26:44.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.046+0000 7fd022ab3640 1 -- 172.21.15.67:0/2752151727 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01c06fd60 con 0x7fd01c145ae0 2024-03-29T11:26:44.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.047+0000 7fd022ab3640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:44.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.047+0000 7fd022ab3640 10 monclient(hunting): authenticate will time out at 1264.342914s 2024-03-29T11:26:44.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.047+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074129946 0 0) 0x7fd01c06fd60 con 0x7fd01c145ae0 2024-03-29T11:26:44.041 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.047+0000 7fd0197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.047+0000 7fd0197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.047+0000 7fd0197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.047+0000 7fd0197fa640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:26:44.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.047+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd008004eb0 con 0x7fd01c145ae0 2024-03-29T11:26:44.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.047+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575656836 0 0) 0x7fd01c070f40 con 0x7fd01c0716d0 2024-03-29T11:26:44.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.047+0000 7fd0197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.047+0000 7fd0197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.047+0000 7fd0197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.042 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.047+0000 7fd0197fa640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:26:44.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.047+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd01c06fd60 con 0x7fd01c0716d0 2024-03-29T11:26:44.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1964969312 0 0) 0x7fd01c06fd60 con 0x7fd01c0716d0 2024-03-29T11:26:44.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd008005490 con 0x7fd01c0716d0 2024-03-29T11:26:44.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd014001c50 con 0x7fd01c0716d0 2024-03-29T11:26:44.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:44.043 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 10 monclient(hunting): dump: 2024-03-29T11:26:44.043 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:44.043 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:44.043 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 788842799 0 0) 0x7fd008004eb0 con 0x7fd01c145ae0 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd008005b60 con 0x7fd01c145ae0 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd0100015f0 con 0x7fd01c145ae0 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 10 monclient(hunting): dump: 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3706364633 0 0) 0x7fd008005490 con 0x7fd01c0716d0 2024-03-29T11:26:44.049 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 >> v1:172.21.15.67:6790/0 conn(0x7fd01c145ae0 legacy=0x7fd01c0ab310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 >> v1:172.21.15.82:6789/0 conn(0x7fd01c070ab0 legacy=0x7fd01c134800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 10 monclient: _finish_hunting 0 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 1 monclient: found mon.a 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c070040 con 0x7fd01c0716d0 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 10 monclient: _finish_auth 0 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 10 monclient: _check_auth_tickets 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd0197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd022ab3640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd022ab3640 10 monclient: _renew_subs 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd022ab3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd022ab3640 1 -- 172.21.15.67:0/2752151727 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01c070320 con 0x7fd01c0716d0 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd022ab3640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd022ab3640 10 monclient: _renew_subs 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd022ab3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd022ab3640 1 -- 172.21.15.67:0/2752151727 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01c071b20 con 0x7fd01c0716d0 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.048+0000 7fd022ab3640 1 librados: init done 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.049+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0140020e0 con 0x7fd01c0716d0 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.049+0000 7fd0197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.049+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd014001c50 con 0x7fd01c0716d0 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.049+0000 7fd0197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.049+0000 7fd0197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.049+0000 7fd0197fa640 10 monclient: dump: 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:44.052 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.052 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:44.052 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:44.052 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:44.052 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:44.052 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:44.052 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.049+0000 7fd01a7fc640 4 set_mon_vals no callback set 2024-03-29T11:26:44.052 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.050+0000 7fd022ab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.052 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.050+0000 7fd022ab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.052 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.050+0000 7fd022ab3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:44.052 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.050+0000 7fd022ab3640 1 -- 172.21.15.67:0/2752151727 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcffc005080 con 0x7fd01c0716d0 2024-03-29T11:26:44.053 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.052+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fd014004b60 con 0x7fd01c0716d0 2024-03-29T11:26:44.054 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.052+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fd014084880 con 0x7fd01c0716d0 2024-03-29T11:26:44.054 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.052+0000 7fd0197fa640 10 client.4476.objecter ms_dispatch 0x7fd01c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:44.054 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.052+0000 7fd0197fa640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:44.054 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.052+0000 7fd0197fa640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:44.054 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.052+0000 7fd02084f640 10 monclient: get_auth_request con 0x7fd00806e3d0 auth_method 0 2024-03-29T11:26:44.054 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.052+0000 7fd0197fa640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-29T11:26:44.055 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.053+0000 7fd0197fa640 10 client.4476.objecter ms_handle_connect 0x7fd00806e3d0 2024-03-29T11:26:44.055 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.062+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd0140501c0 con 0x7fd01c0716d0 2024-03-29T11:26:44.055 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.062+0000 7fd0197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.055 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.062+0000 7fd0197fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:44.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.394+0000 7fd022ab3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:44.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.394+0000 7fd022ab3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:44.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.394+0000 7fd022ab3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:44.382 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.394+0000 7fd022ab3640 1 -- 172.21.15.67:0/2752151727 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcffc005330 con 0x7fd01c0716d0 2024-03-29T11:26:44.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.396+0000 7fd0197fa640 1 -- 172.21.15.67:0/2752151727 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fd0140501c0 con 0x7fd01c0716d0 2024-03-29T11:26:44.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.396+0000 7fd0197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:44.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.396+0000 7fd0197fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:44.383 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:26:44.383 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:44.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.400+0000 7fd022ab3640 10 librados: watch_flush enter 2024-03-29T11:26:44.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.401+0000 7fd022ab3640 10 librados: watch_flush exit 2024-03-29T11:26:44.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.401+0000 7fd022ab3640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-29T11:26:44.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.401+0000 7fd022ab3640 10 client.4476.objecter successfully canceled tick 2024-03-29T11:26:44.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.401+0000 7fd022ab3640 1 -- 172.21.15.67:0/2752151727 >> v1:172.21.15.67:6816/404787197 conn(0x7fd00806e3d0 legacy=0x7fd008070890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.401+0000 7fd022ab3640 10 monclient: shutdown 2024-03-29T11:26:44.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.401+0000 7fd022ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:44.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.401+0000 7fd022ab3640 1 -- 172.21.15.67:0/2752151727 >> v1:172.21.15.67:6789/0 conn(0x7fd01c0716d0 legacy=0x7fd01c134f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.401+0000 7fd022ab3640 1 -- 172.21.15.67:0/2752151727 shutdown_connections 2024-03-29T11:26:44.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.401+0000 7fd022ab3640 1 -- 172.21.15.67:0/2752151727 >> 172.21.15.67:0/2752151727 conn(0x7fd01c06b660 msgr2=0x7fd01c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:44.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.401+0000 7fd022ab3640 1 -- 172.21.15.67:0/2752151727 shutdown_connections 2024-03-29T11:26:44.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.401+0000 7fd022ab3640 1 -- 172.21.15.67:0/2752151727 wait complete. 2024-03-29T11:26:44.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:44.401+0000 7fd022ab3640 1 librados: shutdown 2024-03-29T11:26:44.406 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:51.218 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:51.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.381+0000 7ff2a2dc4640 10 monclient: get_monmap_and_config 2024-03-29T11:26:51.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.381+0000 7ff2a2dc4640 10 monclient: build_initial_monmap 2024-03-29T11:26:51.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.381+0000 7ff2a2dc4640 10 monclient: monmap: 2024-03-29T11:26:51.368 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:51.369 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:51.369 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:26:51.382592+0000 2024-03-29T11:26:51.369 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:26:51.382592+0000 2024-03-29T11:26:51.369 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:51.369 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:51.369 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:51.369 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:26:51.369 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:26:51.369 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:51.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.384+0000 7ff2a2dc4640 1 Processor -- start 2024-03-29T11:26:51.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.384+0000 7ff2a2dc4640 1 -- start start 2024-03-29T11:26:51.373 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.384+0000 7ff2a2dc4640 10 monclient: init 2024-03-29T11:26:51.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.385+0000 7ff2a2dc4640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:51.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.385+0000 7ff2a2dc4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:51.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.385+0000 7ff2a2dc4640 10 monclient(hunting): picked mon.noname-c con 0x7ff29c0b1f60 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:51.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.385+0000 7ff2a2dc4640 10 monclient(hunting): picked mon.noname-b con 0x7ff29c071eb0 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:51.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.385+0000 7ff2a2dc4640 10 monclient(hunting): picked mon.noname-a con 0x7ff29c143d10 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:51.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.385+0000 7ff2a2dc4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c06ef80 con 0x7ff29c143d10 2024-03-29T11:26:51.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.385+0000 7ff2a2dc4640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c06f260 con 0x7ff29c0b1f60 2024-03-29T11:26:51.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.385+0000 7ff2a2dc4640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c0b40c0 con 0x7ff29c071eb0 2024-03-29T11:26:51.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.385+0000 7ff2a2dc4640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:51.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.385+0000 7ff2a2dc4640 10 monclient(hunting): authenticate will time out at 1271.681207s 2024-03-29T11:26:51.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.390+0000 7ff29bfff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7ff29c071eb0 0x7ff29c146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:53115/0 (socket says 172.21.15.67:53115) 2024-03-29T11:26:51.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.391+0000 7ff29bfff640 1 -- 172.21.15.67:0/2983919954 learned_addr learned my addr 172.21.15.67:0/2983919954 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:51.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 1 -- 172.21.15.67:0/2983919954 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681340162 0 0) 0x7ff29c0b40c0 con 0x7ff29c071eb0 2024-03-29T11:26:51.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:51.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:51.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:51.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:26:51.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 1 -- 172.21.15.67:0/2983919954 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff284003520 con 0x7ff29c071eb0 2024-03-29T11:26:51.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 1 -- 172.21.15.67:0/2983919954 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1549913315 0 0) 0x7ff284003520 con 0x7ff29c071eb0 2024-03-29T11:26:51.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 1 -- 172.21.15.67:0/2983919954 >> v1:172.21.15.82:6789/0 conn(0x7ff29c0b1f60 legacy=0x7ff29c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:51.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 1 -- 172.21.15.67:0/2983919954 >> v1:172.21.15.67:6789/0 conn(0x7ff29c143d10 legacy=0x7ff29c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:51.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:51.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:26:51.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:51.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:26:51.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 1 -- 172.21.15.67:0/2983919954 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29c155d70 con 0x7ff29c071eb0 2024-03-29T11:26:51.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:51.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:51.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:51.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 1 -- 172.21.15.67:0/2983919954 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff294001a10 con 0x7ff29c071eb0 2024-03-29T11:26:51.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:51.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:51.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff29b7fe640 10 monclient: dump: 2024-03-29T11:26:51.383 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:51.383 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff2a2dc4640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.392+0000 7ff2a2dc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.393+0000 7ff29b7fe640 1 -- 172.21.15.67:0/2983919954 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff294001b70 con 0x7ff29c071eb0 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.393+0000 7ff29b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.393+0000 7ff29b7fe640 1 -- 172.21.15.67:0/2983919954 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff294001fc0 con 0x7ff29c071eb0 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.393+0000 7ff29b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:51.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.393+0000 7ff29b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:51.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.393+0000 7ff29b7fe640 10 monclient: dump: 2024-03-29T11:26:51.385 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:51.385 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:51.385 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:51.385 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:51.385 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:51.385 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:51.385 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:51.385 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:51.385 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:51.385 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:51.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.393+0000 7ff2a2dc4640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:51.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.393+0000 7ff2a2dc4640 4 set_mon_vals no callback set 2024-03-29T11:26:51.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.394+0000 7ff2a2dc4640 10 monclient: shutdown 2024-03-29T11:26:51.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.394+0000 7ff2a2dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:51.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.394+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2983919954 >> v1:172.21.15.67:6790/0 conn(0x7ff29c071eb0 legacy=0x7ff29c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:51.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.397+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2983919954 shutdown_connections 2024-03-29T11:26:51.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.397+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2983919954 >> 172.21.15.67:0/2983919954 conn(0x7ff29c066a00 msgr2=0x7ff29c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:51.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.397+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2983919954 shutdown_connections 2024-03-29T11:26:51.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.397+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2983919954 wait complete. 2024-03-29T11:26:51.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.398+0000 7ff2a2dc4640 10 monclient: build_initial_monmap 2024-03-29T11:26:51.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.398+0000 7ff2a2dc4640 10 monclient: monmap: 2024-03-29T11:26:51.387 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:51.387 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.398+0000 7ff2a2dc4640 1 librados: starting msgr at 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.398+0000 7ff2a2dc4640 1 librados: starting objecter 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.398+0000 7ff2a2dc4640 1 Processor -- start 2024-03-29T11:26:51.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.399+0000 7ff2a2dc4640 1 -- start start 2024-03-29T11:26:51.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.399+0000 7ff2a2dc4640 1 librados: setting wanted keys 2024-03-29T11:26:51.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.399+0000 7ff2a2dc4640 1 librados: calling monclient init 2024-03-29T11:26:51.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.399+0000 7ff2a2dc4640 10 monclient: init 2024-03-29T11:26:51.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.400+0000 7ff2a2dc4640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:51.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.400+0000 7ff2a2dc4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:51.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.400+0000 7ff2a2dc4640 10 monclient(hunting): picked mon.noname-c con 0x7ff29c0b1f60 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:51.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.400+0000 7ff2a2dc4640 10 monclient(hunting): picked mon.noname-a con 0x7ff29c143d10 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.400+0000 7ff2a2dc4640 10 monclient(hunting): picked mon.noname-b con 0x7ff29c149350 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.400+0000 7ff2a2dc4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c149b30 con 0x7ff29c143d10 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.400+0000 7ff2a2dc4640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c13ba60 con 0x7ff29c149350 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.400+0000 7ff2a2dc4640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c14b210 con 0x7ff29c0b1f60 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.400+0000 7ff2a2dc4640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.400+0000 7ff2a2dc4640 10 monclient(hunting): authenticate will time out at 1271.696192s 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.403+0000 7ff29bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff29c143d10 0x7ff29c13ac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47219/0 (socket says 172.21.15.67:47219) 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.403+0000 7ff29bfff640 1 -- 172.21.15.67:0/2055402718 learned_addr learned my addr 172.21.15.67:0/2055402718 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.403+0000 7ff2997fa640 10 client.?.objecter ms_handle_connect 0x7ff29c143d10 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.403+0000 7ff2997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.403+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437898585 0 0) 0x7ff29c149b30 con 0x7ff29c143d10 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.403+0000 7ff2997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.403+0000 7ff2997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.403+0000 7ff2997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.403+0000 7ff2997fa640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.403+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff288004eb0 con 0x7ff29c143d10 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.404+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 123570313 0 0) 0x7ff288004eb0 con 0x7ff29c143d10 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.404+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff29c149b30 con 0x7ff29c143d10 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.404+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff294001f40 con 0x7ff29c143d10 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.404+0000 7ff2997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.404+0000 7ff2997fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.404+0000 7ff2997fa640 10 monclient(hunting): dump: 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:51.396 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:51.396 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:51.396 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:51.396 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:51.396 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:51.396 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.404+0000 7ff2997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:51.396 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.407+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 592777867 0 0) 0x7ff29c149b30 con 0x7ff29c143d10 2024-03-29T11:26:51.396 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.407+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 >> v1:172.21.15.67:6790/0 conn(0x7ff29c0b1f60 legacy=0x7ff29c147380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:51.396 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 >> v1:172.21.15.82:6789/0 conn(0x7ff29c149350 legacy=0x7ff29c13b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:51.396 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 10 monclient: _finish_hunting 0 2024-03-29T11:26:51.396 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 1 monclient: found mon.a 2024-03-29T11:26:51.396 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29c14b4f0 con 0x7ff29c143d10 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 10 monclient: _finish_auth 0 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 10 monclient: _check_auth_tickets 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2a2dc4640 5 monclient: authenticate success, global_id 4482 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2a2dc4640 10 monclient: _renew_subs 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2a2dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2055402718 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29c1497c0 con 0x7ff29c143d10 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2a2dc4640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2a2dc4640 10 monclient: _renew_subs 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2a2dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2055402718 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29c070fd0 con 0x7ff29c143d10 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2a2dc4640 1 librados: init done 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff294004f70 con 0x7ff29c143d10 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ff294005280 con 0x7ff29c143d10 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff2997fa640 10 monclient: dump: 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:51.399 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:51.399 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.408+0000 7ff29affd640 4 set_mon_vals no callback set 2024-03-29T11:26:51.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.410+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7ff29401b250 con 0x7ff29c143d10 2024-03-29T11:26:51.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.411+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7ff294054e50 con 0x7ff29c143d10 2024-03-29T11:26:51.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.411+0000 7ff2997fa640 10 client.4482.objecter ms_dispatch 0x7ff29c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:51.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.411+0000 7ff2997fa640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:51.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.411+0000 7ff2997fa640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:51.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.411+0000 7ff2997fa640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-29T11:26:51.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.412+0000 7ff2a0b60640 10 monclient: get_auth_request con 0x7ff28806db10 auth_method 0 2024-03-29T11:26:51.400 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.412+0000 7ff2997fa640 10 client.4482.objecter ms_handle_connect 0x7ff28806db10 2024-03-29T11:26:51.401 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.412+0000 7ff2a2dc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:51.401 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.412+0000 7ff2a2dc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:51.401 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.412+0000 7ff2a2dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:51.401 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.412+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2055402718 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27c005080 con 0x7ff29c143d10 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.422+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff2940510f0 con 0x7ff29c143d10 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.423+0000 7ff2997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.423+0000 7ff2997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:51.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.797+0000 7ff2a2dc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:51.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.797+0000 7ff2a2dc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:51.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.797+0000 7ff2a2dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:51.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.797+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2055402718 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff27c005330 con 0x7ff29c143d10 2024-03-29T11:26:51.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.799+0000 7ff2997fa640 1 -- 172.21.15.67:0/2055402718 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7ff294051290 con 0x7ff29c143d10 2024-03-29T11:26:51.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.799+0000 7ff2997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:51.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.799+0000 7ff2997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:51.786 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:26:51.786 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:51.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.802+0000 7ff2a2dc4640 10 librados: watch_flush enter 2024-03-29T11:26:51.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.802+0000 7ff2a2dc4640 10 librados: watch_flush exit 2024-03-29T11:26:51.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.802+0000 7ff2a2dc4640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-29T11:26:51.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.802+0000 7ff2a2dc4640 10 client.4482.objecter successfully canceled tick 2024-03-29T11:26:51.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.802+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2055402718 >> v1:172.21.15.67:6816/404787197 conn(0x7ff28806db10 legacy=0x7ff28806ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:51.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.802+0000 7ff2a2dc4640 10 monclient: shutdown 2024-03-29T11:26:51.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.802+0000 7ff2a2dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:51.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.802+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2055402718 >> v1:172.21.15.67:6789/0 conn(0x7ff29c143d10 legacy=0x7ff29c13ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:51.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.803+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2055402718 shutdown_connections 2024-03-29T11:26:51.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.803+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2055402718 >> 172.21.15.67:0/2055402718 conn(0x7ff29c066a00 msgr2=0x7ff29c0b7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:51.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.803+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2055402718 shutdown_connections 2024-03-29T11:26:51.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.803+0000 7ff2a2dc4640 1 -- 172.21.15.67:0/2055402718 wait complete. 2024-03-29T11:26:51.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:51.803+0000 7ff2a2dc4640 1 librados: shutdown 2024-03-29T11:26:51.805 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:58.615 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.788+0000 7fbadce0d640 10 monclient: get_monmap_and_config 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.788+0000 7fbadce0d640 10 monclient: build_initial_monmap 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.789+0000 7fbadce0d640 10 monclient: monmap: 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:26:58.789794+0000 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:26:58.789794+0000 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:26:58.777 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:58.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.790+0000 7fbadce0d640 1 Processor -- start 2024-03-29T11:26:58.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.790+0000 7fbadce0d640 1 -- start start 2024-03-29T11:26:58.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.790+0000 7fbadce0d640 10 monclient: init 2024-03-29T11:26:58.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.791+0000 7fbadce0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.791+0000 7fbadce0d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.791+0000 7fbadce0d640 10 monclient(hunting): picked mon.noname-c con 0x7fbad81457f0 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.791+0000 7fbadce0d640 10 monclient(hunting): picked mon.noname-a con 0x7fbad813f650 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.791+0000 7fbadce0d640 10 monclient(hunting): picked mon.noname-b con 0x7fbad80a9d70 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.791+0000 7fbadce0d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad80aaef0 con 0x7fbad813f650 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.791+0000 7fbadce0d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad8145bd0 con 0x7fbad81457f0 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.791+0000 7fbadce0d640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad8145eb0 con 0x7fbad80a9d70 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.791+0000 7fbadce0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:58.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.791+0000 7fbadce0d640 10 monclient(hunting): authenticate will time out at 1279.087140s 2024-03-29T11:26:58.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.796+0000 7fbad659c640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fbad81457f0 0x7fbad813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:44033/0 (socket says 172.21.15.67:44033) 2024-03-29T11:26:58.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.796+0000 7fbad659c640 1 -- 172.21.15.67:0/1436444713 learned_addr learned my addr 172.21.15.67:0/1436444713 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:58.787 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.796+0000 7fbad559a640 1 -- 172.21.15.67:0/1436444713 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19633240 0 0) 0x7fbad8145bd0 con 0x7fbad81457f0 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.796+0000 7fbad559a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.796+0000 7fbad559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.797+0000 7fbad559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.797+0000 7fbad559a640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.797+0000 7fbad559a640 1 -- 172.21.15.67:0/1436444713 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbac0003520 con 0x7fbad81457f0 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.799+0000 7fbad559a640 1 -- 172.21.15.67:0/1436444713 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513076487 0 0) 0x7fbad80aaef0 con 0x7fbad813f650 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.799+0000 7fbad559a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.799+0000 7fbad559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.799+0000 7fbad559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.799+0000 7fbad559a640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.799+0000 7fbad559a640 1 -- 172.21.15.67:0/1436444713 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad8145bd0 con 0x7fbad813f650 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.799+0000 7fbad559a640 1 -- 172.21.15.67:0/1436444713 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523482801 0 0) 0x7fbad8145eb0 con 0x7fbad80a9d70 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:58.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:58.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:58.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:26:58.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 1 -- 172.21.15.67:0/1436444713 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad80aaef0 con 0x7fbad80a9d70 2024-03-29T11:26:58.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 1 -- 172.21.15.67:0/1436444713 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1683943028 0 0) 0x7fbac0003520 con 0x7fbad81457f0 2024-03-29T11:26:58.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 1 -- 172.21.15.67:0/1436444713 >> v1:172.21.15.67:6790/0 conn(0x7fbad80a9d70 legacy=0x7fbad814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:58.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 1 -- 172.21.15.67:0/1436444713 >> v1:172.21.15.67:6789/0 conn(0x7fbad813f650 legacy=0x7fbad80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:58.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 10 monclient: _finish_hunting 0 2024-03-29T11:26:58.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 1 monclient: found mon.noname-c 2024-03-29T11:26:58.789 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:58.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:26:58.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 1 -- 172.21.15.67:0/1436444713 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad8146340 con 0x7fbad81457f0 2024-03-29T11:26:58.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 10 monclient: _finish_auth 0 2024-03-29T11:26:58.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 10 monclient: _check_auth_tickets 2024-03-29T11:26:58.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:58.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 1 -- 172.21.15.67:0/1436444713 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fbac8001610 con 0x7fbad81457f0 2024-03-29T11:26:58.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:58.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:58.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.800+0000 7fbad559a640 10 monclient: dump: 2024-03-29T11:26:58.790 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:58.790 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:58.790 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:58.790 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:58.791 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:58.791 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:58.791 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:58.791 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:58.791 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:58.791 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:58.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.803+0000 7fbad559a640 1 -- 172.21.15.67:0/1436444713 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbac8001770 con 0x7fbad81457f0 2024-03-29T11:26:58.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.803+0000 7fbad559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:58.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.803+0000 7fbad559a640 1 -- 172.21.15.67:0/1436444713 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fbac8001bc0 con 0x7fbad81457f0 2024-03-29T11:26:58.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.803+0000 7fbad559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:58.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.803+0000 7fbad559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:58.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.803+0000 7fbad559a640 10 monclient: dump: 2024-03-29T11:26:58.793 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:58.793 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:58.793 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:58.793 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:58.793 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:58.793 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:58.793 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:58.793 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:58.793 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:58.793 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:58.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.804+0000 7fbadce0d640 5 monclient: authenticate success, global_id 4441 2024-03-29T11:26:58.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.804+0000 7fbadce0d640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:58.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.804+0000 7fbadce0d640 4 set_mon_vals no callback set 2024-03-29T11:26:58.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.805+0000 7fbadce0d640 10 monclient: shutdown 2024-03-29T11:26:58.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.805+0000 7fbadce0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:58.794 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.805+0000 7fbadce0d640 1 -- 172.21.15.67:0/1436444713 >> v1:172.21.15.82:6789/0 conn(0x7fbad81457f0 legacy=0x7fbad813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:58.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.806+0000 7fbadce0d640 1 -- 172.21.15.67:0/1436444713 shutdown_connections 2024-03-29T11:26:58.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.806+0000 7fbadce0d640 1 -- 172.21.15.67:0/1436444713 >> 172.21.15.67:0/1436444713 conn(0x7fbad8066a00 msgr2=0x7fbad8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:58.795 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.808+0000 7fbadce0d640 1 -- 172.21.15.67:0/1436444713 shutdown_connections 2024-03-29T11:26:58.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.808+0000 7fbadce0d640 1 -- 172.21.15.67:0/1436444713 wait complete. 2024-03-29T11:26:58.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.809+0000 7fbadce0d640 10 monclient: build_initial_monmap 2024-03-29T11:26:58.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.809+0000 7fbadce0d640 10 monclient: monmap: 2024-03-29T11:26:58.797 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:26:58.797 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:58.797 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:26:58.797 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:26:58.797 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:58.797 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:58.798 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:26:58.798 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:26:58.798 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:26:58.798 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:58.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.810+0000 7fbadce0d640 1 librados: starting msgr at 2024-03-29T11:26:58.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.810+0000 7fbadce0d640 1 librados: starting objecter 2024-03-29T11:26:58.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.810+0000 7fbadce0d640 1 Processor -- start 2024-03-29T11:26:58.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.810+0000 7fbadce0d640 1 -- start start 2024-03-29T11:26:58.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.810+0000 7fbadce0d640 1 librados: setting wanted keys 2024-03-29T11:26:58.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.810+0000 7fbadce0d640 1 librados: calling monclient init 2024-03-29T11:26:58.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.810+0000 7fbadce0d640 10 monclient: init 2024-03-29T11:26:58.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.811+0000 7fbadce0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:58.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.811+0000 7fbadce0d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:58.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.813+0000 7fbadce0d640 10 monclient(hunting): picked mon.noname-a con 0x7fbad80a9d70 addr v1:172.21.15.67:6789/0 2024-03-29T11:26:58.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.814+0000 7fbadce0d640 10 monclient(hunting): picked mon.noname-b con 0x7fbad813f650 addr v1:172.21.15.82:6789/0 2024-03-29T11:26:58.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.814+0000 7fbadce0d640 10 monclient(hunting): picked mon.noname-c con 0x7fbad81457f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:26:58.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.814+0000 7fbadce0d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad8070b80 con 0x7fbad80a9d70 2024-03-29T11:26:58.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.814+0000 7fbadce0d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad806fed0 con 0x7fbad813f650 2024-03-29T11:26:58.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.814+0000 7fbadce0d640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad8070060 con 0x7fbad81457f0 2024-03-29T11:26:58.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.814+0000 7fbadce0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:58.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.814+0000 7fbadce0d640 10 monclient(hunting): authenticate will time out at 1279.110754s 2024-03-29T11:26:58.807 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.817+0000 7fbad659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbad80a9d70 0x7fbad8070340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50377/0 (socket says 172.21.15.67:50377) 2024-03-29T11:26:58.808 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.817+0000 7fbad659c640 1 -- 172.21.15.67:0/963828715 learned_addr learned my addr 172.21.15.67:0/963828715 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:26:58.808 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.817+0000 7fbac6ffd640 10 client.?.objecter ms_handle_connect 0x7fbad80a9d70 2024-03-29T11:26:58.808 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.817+0000 7fbac6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:58.808 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402272261 0 0) 0x7fbad8070b80 con 0x7fbad80a9d70 2024-03-29T11:26:58.808 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:58.808 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:58.808 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:58.808 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:26:58.808 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab8004eb0 con 0x7fbad80a9d70 2024-03-29T11:26:58.808 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1108231594 0 0) 0x7fbab8004eb0 con 0x7fbad80a9d70 2024-03-29T11:26:58.808 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbad8070b80 con 0x7fbad80a9d70 2024-03-29T11:26:58.808 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fbac8001b80 con 0x7fbad80a9d70 2024-03-29T11:26:58.808 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:58.808 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:58.809 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:58.809 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:58.809 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:58.809 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:58.809 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:58.809 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:58.809 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:58.809 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:58.809 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:58.809 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:58.810 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:58.810 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:58.810 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 775733328 0 0) 0x7fbad8070b80 con 0x7fbad80a9d70 2024-03-29T11:26:58.810 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 >> v1:172.21.15.67:6790/0 conn(0x7fbad81457f0 legacy=0x7fbad81df6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:58.810 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 >> v1:172.21.15.82:6789/0 conn(0x7fbad813f650 legacy=0x7fbad81defc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:58.810 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:58.810 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 1 monclient: found mon.a 2024-03-29T11:26:58.810 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:58.810 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:58.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad8142210 con 0x7fbad80a9d70 2024-03-29T11:26:58.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:58.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:58.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:58.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbac8004a90 con 0x7fbad80a9d70 2024-03-29T11:26:58.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.818+0000 7fbac6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:58.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.819+0000 7fbad4d99640 4 set_mon_vals no callback set 2024-03-29T11:26:58.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.820+0000 7fbadce0d640 5 monclient: authenticate success, global_id 4488 2024-03-29T11:26:58.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.820+0000 7fbadce0d640 10 monclient: _renew_subs 2024-03-29T11:26:58.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.820+0000 7fbadce0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:58.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.820+0000 7fbadce0d640 1 -- 172.21.15.67:0/963828715 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad81424c0 con 0x7fbad80a9d70 2024-03-29T11:26:58.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.820+0000 7fbadce0d640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:58.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.820+0000 7fbadce0d640 10 monclient: _renew_subs 2024-03-29T11:26:58.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.820+0000 7fbadce0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:58.812 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.820+0000 7fbadce0d640 1 -- 172.21.15.67:0/963828715 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad81e75a0 con 0x7fbad80a9d70 2024-03-29T11:26:58.812 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.820+0000 7fbadce0d640 1 librados: init done 2024-03-29T11:26:58.813 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.821+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fbac8004e60 con 0x7fbad80a9d70 2024-03-29T11:26:58.813 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.821+0000 7fbac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:58.813 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.821+0000 7fbac6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:58.813 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.821+0000 7fbac6ffd640 10 monclient: dump: 2024-03-29T11:26:58.813 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:26:58.813 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:26:58.814 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:58.814 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:26:58.814 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:58.814 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:26:58.814 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:26:58.814 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:26:58.814 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:26:58.814 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:26:58.814 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.822+0000 7fbadce0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:58.814 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.822+0000 7fbadce0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:58.814 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.822+0000 7fbadce0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:58.814 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.822+0000 7fbadce0d640 1 -- 172.21.15.67:0/963828715 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad8075f10 con 0x7fbad80a9d70 2024-03-29T11:26:58.814 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.823+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fbac801ae30 con 0x7fbad80a9d70 2024-03-29T11:26:58.815 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.823+0000 7fbad5d9b640 10 monclient: get_auth_request con 0x7fbab806dca0 auth_method 0 2024-03-29T11:26:58.815 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.824+0000 7fbac6ffd640 10 client.4488.objecter ms_handle_connect 0x7fbab806dca0 2024-03-29T11:26:58.815 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.824+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fbac8084b30 con 0x7fbad80a9d70 2024-03-29T11:26:58.815 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.824+0000 7fbac6ffd640 10 client.4488.objecter ms_dispatch 0x7fbad8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:58.815 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.824+0000 7fbac6ffd640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:58.815 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.824+0000 7fbac6ffd640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:58.815 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.824+0000 7fbac6ffd640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-29T11:26:58.822 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.832+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbac8050410 con 0x7fbad80a9d70 2024-03-29T11:26:58.822 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.832+0000 7fbac6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:58.822 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:58.832+0000 7fbac6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:59.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.166+0000 7fbadce0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.166+0000 7fbadce0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.166+0000 7fbadce0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:26:59.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.166+0000 7fbadce0d640 1 -- 172.21.15.67:0/963828715 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbad80b3680 con 0x7fbad80a9d70 2024-03-29T11:26:59.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.171+0000 7fbac6ffd640 1 -- 172.21.15.67:0/963828715 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fbac8052340 con 0x7fbad80a9d70 2024-03-29T11:26:59.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.171+0000 7fbac6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.171+0000 7fbac6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:59.159 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:26:59.159 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:59.161 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.174+0000 7fbadce0d640 10 librados: watch_flush enter 2024-03-29T11:26:59.161 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.174+0000 7fbadce0d640 10 librados: watch_flush exit 2024-03-29T11:26:59.161 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.174+0000 7fbadce0d640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-29T11:26:59.161 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.174+0000 7fbadce0d640 10 client.4488.objecter successfully canceled tick 2024-03-29T11:26:59.161 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.174+0000 7fbadce0d640 1 -- 172.21.15.67:0/963828715 >> v1:172.21.15.67:6816/404787197 conn(0x7fbab806dca0 legacy=0x7fbab8070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.161 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.174+0000 7fbadce0d640 10 monclient: shutdown 2024-03-29T11:26:59.161 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.174+0000 7fbadce0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:59.162 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.174+0000 7fbadce0d640 1 -- 172.21.15.67:0/963828715 >> v1:172.21.15.67:6789/0 conn(0x7fbad80a9d70 legacy=0x7fbad8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.164 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.177+0000 7fbadce0d640 1 -- 172.21.15.67:0/963828715 shutdown_connections 2024-03-29T11:26:59.164 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.177+0000 7fbadce0d640 1 -- 172.21.15.67:0/963828715 >> 172.21.15.67:0/963828715 conn(0x7fbad8066a00 msgr2=0x7fbad80b4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:59.164 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.177+0000 7fbadce0d640 1 -- 172.21.15.67:0/963828715 shutdown_connections 2024-03-29T11:26:59.164 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.177+0000 7fbadce0d640 1 -- 172.21.15.67:0/963828715 wait complete. 2024-03-29T11:26:59.164 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:26:59.177+0000 7fbadce0d640 1 librados: shutdown 2024-03-29T11:26:59.181 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:05.989 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:06.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.164+0000 7ffafae46640 10 monclient: get_monmap_and_config 2024-03-29T11:27:06.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.164+0000 7ffafae46640 10 monclient: build_initial_monmap 2024-03-29T11:27:06.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.164+0000 7ffafae46640 10 monclient: monmap: 2024-03-29T11:27:06.152 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:06.152 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:06.153 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:27:06.165948+0000 2024-03-29T11:27:06.153 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:27:06.165948+0000 2024-03-29T11:27:06.153 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:06.153 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:06.153 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:06.153 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:27:06.153 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:27:06.153 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:06.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.165+0000 7ffafae46640 1 Processor -- start 2024-03-29T11:27:06.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.166+0000 7ffafae46640 1 -- start start 2024-03-29T11:27:06.154 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.166+0000 7ffafae46640 10 monclient: init 2024-03-29T11:27:06.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.167+0000 7ffafae46640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:06.155 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.167+0000 7ffafae46640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:06.162 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.167+0000 7ffafae46640 10 monclient(hunting): picked mon.noname-b con 0x7ffaf40b4920 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:06.162 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.167+0000 7ffafae46640 10 monclient(hunting): picked mon.noname-c con 0x7ffaf40b4200 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:06.162 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.167+0000 7ffafae46640 10 monclient(hunting): picked mon.noname-a con 0x7ffaf413f9a0 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:06.162 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.167+0000 7ffafae46640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf40b0930 con 0x7ffaf413f9a0 2024-03-29T11:27:06.162 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.167+0000 7ffafae46640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf40af460 con 0x7ffaf40b4200 2024-03-29T11:27:06.162 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.167+0000 7ffafae46640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf40aeda0 con 0x7ffaf40b4920 2024-03-29T11:27:06.162 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.167+0000 7ffafae46640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:06.162 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.167+0000 7ffafae46640 10 monclient(hunting): authenticate will time out at 1286.464477s 2024-03-29T11:27:06.163 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.174+0000 7ffaf9e44640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7ffaf40b4920 0x7ffaf413f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:37283/0 (socket says 172.21.15.67:37283) 2024-03-29T11:27:06.163 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.174+0000 7ffaf9e44640 1 -- 172.21.15.67:0/3218209467 learned_addr learned my addr 172.21.15.67:0/3218209467 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:06.163 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.175+0000 7ffaf8e42640 1 -- 172.21.15.67:0/3218209467 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467020792 0 0) 0x7ffaf40af460 con 0x7ffaf40b4200 2024-03-29T11:27:06.165 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.175+0000 7ffaf8e42640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.165 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.175+0000 7ffaf8e42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.165 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.176+0000 7ffaf8e42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.165 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.176+0000 7ffaf8e42640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:27:06.165 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.176+0000 7ffaf8e42640 1 -- 172.21.15.67:0/3218209467 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae0003520 con 0x7ffaf40b4200 2024-03-29T11:27:06.165 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.176+0000 7ffaf8e42640 1 -- 172.21.15.67:0/3218209467 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320173744 0 0) 0x7ffaf40aeda0 con 0x7ffaf40b4920 2024-03-29T11:27:06.165 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.176+0000 7ffaf8e42640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.165 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.176+0000 7ffaf8e42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.176+0000 7ffaf8e42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.176+0000 7ffaf8e42640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:27:06.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.176+0000 7ffaf8e42640 1 -- 172.21.15.67:0/3218209467 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf40af460 con 0x7ffaf40b4920 2024-03-29T11:27:06.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.176+0000 7ffaf8e42640 1 -- 172.21.15.67:0/3218209467 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349073749 0 0) 0x7ffaf40b0930 con 0x7ffaf413f9a0 2024-03-29T11:27:06.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.176+0000 7ffaf8e42640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.176+0000 7ffaf8e42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.177+0000 7ffaf8e42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.177+0000 7ffaf8e42640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:27:06.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.177+0000 7ffaf8e42640 1 -- 172.21.15.67:0/3218209467 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf40aeda0 con 0x7ffaf413f9a0 2024-03-29T11:27:06.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 1 -- 172.21.15.67:0/3218209467 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 914170811 0 0) 0x7ffaf40af460 con 0x7ffaf40b4920 2024-03-29T11:27:06.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 1 -- 172.21.15.67:0/3218209467 >> v1:172.21.15.82:6789/0 conn(0x7ffaf40b4200 legacy=0x7ffaf40b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 1 -- 172.21.15.67:0/3218209467 >> v1:172.21.15.67:6789/0 conn(0x7ffaf413f9a0 legacy=0x7ffaf414d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 10 monclient: _finish_hunting 0 2024-03-29T11:27:06.169 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 1 monclient: found mon.noname-b 2024-03-29T11:27:06.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:06.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:27:06.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 1 -- 172.21.15.67:0/3218209467 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf40af980 con 0x7ffaf40b4920 2024-03-29T11:27:06.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 10 monclient: _finish_auth 0 2024-03-29T11:27:06.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 10 monclient: _check_auth_tickets 2024-03-29T11:27:06.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:06.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 1 -- 172.21.15.67:0/3218209467 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ffae80015f0 con 0x7ffaf40b4920 2024-03-29T11:27:06.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:06.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffaf8e42640 10 monclient: dump: 2024-03-29T11:27:06.171 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:06.171 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:06.171 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:06.172 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:06.172 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.172 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:06.172 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:06.172 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:06.172 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:06.172 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:06.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffafae46640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:27:06.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.180+0000 7ffafae46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:06.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.181+0000 7ffaf8e42640 1 -- 172.21.15.67:0/3218209467 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffae8001810 con 0x7ffaf40b4920 2024-03-29T11:27:06.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.181+0000 7ffaf8e42640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:06.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.181+0000 7ffaf8e42640 1 -- 172.21.15.67:0/3218209467 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ffae8001ba0 con 0x7ffaf40b4920 2024-03-29T11:27:06.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.181+0000 7ffaf8e42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.175 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.181+0000 7ffaf8e42640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:06.175 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.181+0000 7ffaf8e42640 10 monclient: dump: 2024-03-29T11:27:06.175 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:06.175 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:06.175 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:06.175 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:06.175 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.175 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:06.175 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:06.175 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:06.175 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:06.175 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:06.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.181+0000 7ffafae46640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:06.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.181+0000 7ffafae46640 4 set_mon_vals no callback set 2024-03-29T11:27:06.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.182+0000 7ffafae46640 10 monclient: shutdown 2024-03-29T11:27:06.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.182+0000 7ffafae46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:06.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.182+0000 7ffafae46640 1 -- 172.21.15.67:0/3218209467 >> v1:172.21.15.67:6790/0 conn(0x7ffaf40b4920 legacy=0x7ffaf413f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.185+0000 7ffafae46640 1 -- 172.21.15.67:0/3218209467 shutdown_connections 2024-03-29T11:27:06.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.185+0000 7ffafae46640 1 -- 172.21.15.67:0/3218209467 >> 172.21.15.67:0/3218209467 conn(0x7ffaf406d230 msgr2=0x7ffaf4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:06.177 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.189+0000 7ffafae46640 1 -- 172.21.15.67:0/3218209467 shutdown_connections 2024-03-29T11:27:06.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.189+0000 7ffafae46640 1 -- 172.21.15.67:0/3218209467 wait complete. 2024-03-29T11:27:06.182 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.190+0000 7ffafae46640 10 monclient: build_initial_monmap 2024-03-29T11:27:06.182 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.190+0000 7ffafae46640 10 monclient: monmap: 2024-03-29T11:27:06.182 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:06.182 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:06.182 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:27:06.183 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:27:06.183 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:06.183 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:06.183 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:06.183 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:27:06.183 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:27:06.183 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:06.183 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.190+0000 7ffafae46640 1 librados: starting msgr at 2024-03-29T11:27:06.183 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.190+0000 7ffafae46640 1 librados: starting objecter 2024-03-29T11:27:06.183 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.190+0000 7ffafae46640 1 Processor -- start 2024-03-29T11:27:06.183 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.191+0000 7ffafae46640 1 -- start start 2024-03-29T11:27:06.183 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.191+0000 7ffafae46640 1 librados: setting wanted keys 2024-03-29T11:27:06.183 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.191+0000 7ffafae46640 1 librados: calling monclient init 2024-03-29T11:27:06.183 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.191+0000 7ffafae46640 10 monclient: init 2024-03-29T11:27:06.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.191+0000 7ffafae46640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:06.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.191+0000 7ffafae46640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:06.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.191+0000 7ffafae46640 10 monclient(hunting): picked mon.noname-a con 0x7ffaf40b4200 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:06.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.191+0000 7ffafae46640 10 monclient(hunting): picked mon.noname-c con 0x7ffaf40b4920 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:06.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.191+0000 7ffafae46640 10 monclient(hunting): picked mon.noname-b con 0x7ffaf413f9a0 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:06.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.191+0000 7ffafae46640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf40ae590 con 0x7ffaf40b4200 2024-03-29T11:27:06.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.191+0000 7ffafae46640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf40ae100 con 0x7ffaf413f9a0 2024-03-29T11:27:06.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.191+0000 7ffafae46640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf40ac120 con 0x7ffaf40b4920 2024-03-29T11:27:06.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.191+0000 7ffafae46640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:06.184 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.191+0000 7ffafae46640 10 monclient(hunting): authenticate will time out at 1286.488749s 2024-03-29T11:27:06.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.197+0000 7ffaf9643640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7ffaf40b4920 0x7ffaf4145810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:37393/0 (socket says 172.21.15.67:37393) 2024-03-29T11:27:06.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.197+0000 7ffaf9643640 1 -- 172.21.15.67:0/4111151077 learned_addr learned my addr 172.21.15.67:0/4111151077 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:06.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.198+0000 7ffae67fc640 10 client.?.objecter ms_handle_connect 0x7ffaf40b4920 2024-03-29T11:27:06.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.198+0000 7ffae67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:06.190 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.199+0000 7ffae67fc640 10 client.?.objecter ms_handle_connect 0x7ffaf413f9a0 2024-03-29T11:27:06.190 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.199+0000 7ffae67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:06.190 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.200+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400657885 0 0) 0x7ffaf40ae100 con 0x7ffaf413f9a0 2024-03-29T11:27:06.190 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.200+0000 7ffae67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.190 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.200+0000 7ffae67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.190 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.200+0000 7ffae67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.190 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.200+0000 7ffae67fc640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:27:06.190 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.200+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad8004eb0 con 0x7ffaf413f9a0 2024-03-29T11:27:06.190 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.200+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 817618509 0 0) 0x7ffad8004eb0 con 0x7ffaf413f9a0 2024-03-29T11:27:06.190 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.200+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffaf40ae100 con 0x7ffaf413f9a0 2024-03-29T11:27:06.191 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.200+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ffaec001b20 con 0x7ffaf413f9a0 2024-03-29T11:27:06.191 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.191 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:06.191 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 10 monclient(hunting): dump: 2024-03-29T11:27:06.191 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:06.191 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:06.191 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:06.191 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:06.191 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.191 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:06.191 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:06.191 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:06.191 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:06.191 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:06.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:06.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3091306133 0 0) 0x7ffaf40ae100 con 0x7ffaf413f9a0 2024-03-29T11:27:06.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 >> v1:172.21.15.67:6790/0 conn(0x7ffaf40b4920 legacy=0x7ffaf4145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 >> v1:172.21.15.67:6789/0 conn(0x7ffaf40b4200 legacy=0x7ffaf40ad460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:06.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 1 monclient: found mon.b 2024-03-29T11:27:06.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:06.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:27:06.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf4140da0 con 0x7ffaf413f9a0 2024-03-29T11:27:06.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:06.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:06.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffae67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:06.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffafae46640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:27:06.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffafae46640 10 monclient: _renew_subs 2024-03-29T11:27:06.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffafae46640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:27:06.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffafae46640 1 -- 172.21.15.67:0/4111151077 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf413fe60 con 0x7ffaf413f9a0 2024-03-29T11:27:06.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffafae46640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:06.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffafae46640 10 monclient: _renew_subs 2024-03-29T11:27:06.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffafae46640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:27:06.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffafae46640 1 -- 172.21.15.67:0/4111151077 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf4145f20 con 0x7ffaf413f9a0 2024-03-29T11:27:06.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.201+0000 7ffafae46640 1 librados: init done 2024-03-29T11:27:06.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.203+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffaec001c80 con 0x7ffaf413f9a0 2024-03-29T11:27:06.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.203+0000 7ffae67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:06.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.203+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7ffaec003b30 con 0x7ffaf413f9a0 2024-03-29T11:27:06.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.203+0000 7ffae67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.203+0000 7ffae67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:06.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.203+0000 7ffae67fc640 10 monclient: dump: 2024-03-29T11:27:06.198 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:06.198 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:06.198 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:06.198 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:06.199 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.199 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:06.199 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:06.199 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:06.199 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:06.199 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:06.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.203+0000 7ffae77fe640 4 set_mon_vals no callback set 2024-03-29T11:27:06.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.205+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7ffaec003c90 con 0x7ffaf413f9a0 2024-03-29T11:27:06.200 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.205+0000 7ffaf9e44640 10 monclient: get_auth_request con 0x7ffad806dbb0 auth_method 0 2024-03-29T11:27:06.200 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.206+0000 7ffae67fc640 10 client.4447.objecter ms_handle_connect 0x7ffad806dbb0 2024-03-29T11:27:06.200 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.206+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7ffaec083b00 con 0x7ffaf413f9a0 2024-03-29T11:27:06.200 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.206+0000 7ffae67fc640 10 client.4447.objecter ms_dispatch 0x7ffaf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:06.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.206+0000 7ffae67fc640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:06.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.206+0000 7ffae67fc640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:06.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.206+0000 7ffae67fc640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-29T11:27:06.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.206+0000 7ffafae46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.206+0000 7ffafae46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.206+0000 7ffafae46640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:27:06.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.206+0000 7ffafae46640 1 -- 172.21.15.67:0/4111151077 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffadc005080 con 0x7ffaf413f9a0 2024-03-29T11:27:06.202 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.213+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffaec04f3e0 con 0x7ffaf413f9a0 2024-03-29T11:27:06.202 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.213+0000 7ffae67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.202 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.213+0000 7ffae67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:06.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.557+0000 7ffafae46640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:06.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.557+0000 7ffafae46640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:06.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.557+0000 7ffafae46640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:27:06.546 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.557+0000 7ffafae46640 1 -- 172.21.15.67:0/4111151077 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffadc005330 con 0x7ffaf413f9a0 2024-03-29T11:27:06.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.559+0000 7ffae67fc640 1 -- 172.21.15.67:0/4111151077 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7ffaec051310 con 0x7ffaf413f9a0 2024-03-29T11:27:06.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.559+0000 7ffae67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:06.547 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.559+0000 7ffae67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:06.547 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:27:06.547 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:06.550 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.562+0000 7ffafae46640 10 librados: watch_flush enter 2024-03-29T11:27:06.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.564+0000 7ffafae46640 10 librados: watch_flush exit 2024-03-29T11:27:06.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.564+0000 7ffafae46640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-29T11:27:06.552 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.564+0000 7ffafae46640 10 client.4447.objecter successfully canceled tick 2024-03-29T11:27:06.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.566+0000 7ffafae46640 1 -- 172.21.15.67:0/4111151077 >> v1:172.21.15.67:6816/404787197 conn(0x7ffad806dbb0 legacy=0x7ffad8070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.566+0000 7ffafae46640 10 monclient: shutdown 2024-03-29T11:27:06.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.566+0000 7ffafae46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:06.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.566+0000 7ffafae46640 1 -- 172.21.15.67:0/4111151077 >> v1:172.21.15.82:6789/0 conn(0x7ffaf413f9a0 legacy=0x7ffaf40aa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.566+0000 7ffafae46640 1 -- 172.21.15.67:0/4111151077 shutdown_connections 2024-03-29T11:27:06.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.566+0000 7ffafae46640 1 -- 172.21.15.67:0/4111151077 >> 172.21.15.67:0/4111151077 conn(0x7ffaf406d230 msgr2=0x7ffaf4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:06.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.567+0000 7ffafae46640 1 -- 172.21.15.67:0/4111151077 shutdown_connections 2024-03-29T11:27:06.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.567+0000 7ffafae46640 1 -- 172.21.15.67:0/4111151077 wait complete. 2024-03-29T11:27:06.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:06.567+0000 7ffafae46640 1 librados: shutdown 2024-03-29T11:27:06.576 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:13.384 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:13.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.543+0000 7f6c856f6640 10 monclient: get_monmap_and_config 2024-03-29T11:27:13.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.543+0000 7f6c856f6640 10 monclient: build_initial_monmap 2024-03-29T11:27:13.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.544+0000 7f6c856f6640 10 monclient: monmap: 2024-03-29T11:27:13.533 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:13.533 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:13.533 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:27:13.545830+0000 2024-03-29T11:27:13.533 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:27:13.545830+0000 2024-03-29T11:27:13.533 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:13.533 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:13.533 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:13.533 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:27:13.533 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:27:13.533 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:13.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.545+0000 7f6c856f6640 1 Processor -- start 2024-03-29T11:27:13.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.545+0000 7f6c856f6640 1 -- start start 2024-03-29T11:27:13.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.545+0000 7f6c856f6640 10 monclient: init 2024-03-29T11:27:13.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.546+0000 7f6c856f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:13.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.546+0000 7f6c856f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:13.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.546+0000 7f6c856f6640 10 monclient(hunting): picked mon.noname-b con 0x7f6c800b1f60 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:13.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.546+0000 7f6c856f6640 10 monclient(hunting): picked mon.noname-a con 0x7f6c80071eb0 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:13.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.546+0000 7f6c856f6640 10 monclient(hunting): picked mon.noname-c con 0x7f6c80143d10 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:13.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.546+0000 7f6c856f6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8006ef80 con 0x7f6c80071eb0 2024-03-29T11:27:13.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.546+0000 7f6c856f6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8006f260 con 0x7f6c80143d10 2024-03-29T11:27:13.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.546+0000 7f6c856f6640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c800b40c0 con 0x7f6c800b1f60 2024-03-29T11:27:13.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.546+0000 7f6c856f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:13.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.546+0000 7f6c856f6640 10 monclient(hunting): authenticate will time out at 1293.843375s 2024-03-29T11:27:13.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.546+0000 7f6c7effd640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f6c800b1f60 0x7f6c80143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:43724/0 (socket says 172.21.15.67:43724) 2024-03-29T11:27:13.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.546+0000 7f6c7effd640 1 -- 172.21.15.67:0/1597464828 learned_addr learned my addr 172.21.15.67:0/1597464828 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:13.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 1 -- 172.21.15.67:0/1597464828 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784437021 0 0) 0x7f6c8006f260 con 0x7f6c80143d10 2024-03-29T11:27:13.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:13.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:13.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:13.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:27:13.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 1 -- 172.21.15.67:0/1597464828 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c68003520 con 0x7f6c80143d10 2024-03-29T11:27:13.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 1 -- 172.21.15.67:0/1597464828 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239162909 0 0) 0x7f6c800b40c0 con 0x7f6c800b1f60 2024-03-29T11:27:13.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:13.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:13.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:13.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:27:13.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 1 -- 172.21.15.67:0/1597464828 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c8006f260 con 0x7f6c800b1f60 2024-03-29T11:27:13.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 1 -- 172.21.15.67:0/1597464828 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020426976 0 0) 0x7f6c8006ef80 con 0x7f6c80071eb0 2024-03-29T11:27:13.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:13.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:13.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:13.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:27:13.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.547+0000 7f6c7dffb640 1 -- 172.21.15.67:0/1597464828 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c800b40c0 con 0x7f6c80071eb0 2024-03-29T11:27:13.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 1 -- 172.21.15.67:0/1597464828 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2770805417 0 0) 0x7f6c8006f260 con 0x7f6c800b1f60 2024-03-29T11:27:13.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 1 -- 172.21.15.67:0/1597464828 >> v1:172.21.15.82:6789/0 conn(0x7f6c80143d10 legacy=0x7f6c80146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:13.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 1 -- 172.21.15.67:0/1597464828 >> v1:172.21.15.67:6789/0 conn(0x7f6c80071eb0 legacy=0x7f6c80146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:13.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:13.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 1 monclient: found mon.noname-b 2024-03-29T11:27:13.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:13.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:27:13.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 1 -- 172.21.15.67:0/1597464828 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c80155d70 con 0x7f6c800b1f60 2024-03-29T11:27:13.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:13.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:13.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:13.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 1 -- 172.21.15.67:0/1597464828 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6c700015f0 con 0x7f6c800b1f60 2024-03-29T11:27:13.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:13.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:13.540 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 10 monclient: dump: 2024-03-29T11:27:13.540 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:13.540 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:13.540 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:13.540 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:13.540 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:13.540 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:13.540 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:13.540 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:13.540 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:13.540 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:13.540 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c856f6640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:27:13.540 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c856f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:13.540 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 1 -- 172.21.15.67:0/1597464828 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c70001810 con 0x7f6c800b1f60 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 1 -- 172.21.15.67:0/1597464828 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6c70001ba0 con 0x7f6c800b1f60 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c7dffb640 10 monclient: dump: 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:13.541 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:13.542 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c856f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:13.542 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.548+0000 7f6c856f6640 4 set_mon_vals no callback set 2024-03-29T11:27:13.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.549+0000 7f6c856f6640 10 monclient: shutdown 2024-03-29T11:27:13.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.549+0000 7f6c856f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:13.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.549+0000 7f6c856f6640 1 -- 172.21.15.67:0/1597464828 >> v1:172.21.15.67:6790/0 conn(0x7f6c800b1f60 legacy=0x7f6c80143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:13.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.549+0000 7f6c856f6640 1 -- 172.21.15.67:0/1597464828 shutdown_connections 2024-03-29T11:27:13.553 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.549+0000 7f6c856f6640 1 -- 172.21.15.67:0/1597464828 >> 172.21.15.67:0/1597464828 conn(0x7f6c80066a00 msgr2=0x7f6c80066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:13.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.549+0000 7f6c856f6640 1 -- 172.21.15.67:0/1597464828 shutdown_connections 2024-03-29T11:27:13.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.549+0000 7f6c856f6640 1 -- 172.21.15.67:0/1597464828 wait complete. 2024-03-29T11:27:13.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.550+0000 7f6c856f6640 10 monclient: build_initial_monmap 2024-03-29T11:27:13.554 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.550+0000 7f6c856f6640 10 monclient: monmap: 2024-03-29T11:27:13.554 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:13.554 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:13.554 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:27:13.555 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:27:13.555 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:13.555 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:13.555 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:13.555 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:27:13.555 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:27:13.555 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:13.555 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.551+0000 7f6c856f6640 1 librados: starting msgr at 2024-03-29T11:27:13.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.551+0000 7f6c856f6640 1 librados: starting objecter 2024-03-29T11:27:13.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.551+0000 7f6c856f6640 1 Processor -- start 2024-03-29T11:27:13.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.551+0000 7f6c856f6640 1 -- start start 2024-03-29T11:27:13.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.551+0000 7f6c856f6640 1 librados: setting wanted keys 2024-03-29T11:27:13.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.551+0000 7f6c856f6640 1 librados: calling monclient init 2024-03-29T11:27:13.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.551+0000 7f6c856f6640 10 monclient: init 2024-03-29T11:27:13.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.552+0000 7f6c856f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:13.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.552+0000 7f6c856f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:13.556 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.552+0000 7f6c856f6640 10 monclient(hunting): picked mon.noname-b con 0x7f6c800b1f60 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:13.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.552+0000 7f6c856f6640 10 monclient(hunting): picked mon.noname-c con 0x7f6c80143d10 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:13.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.552+0000 7f6c7effd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f6c800b1f60 0x7f6c80154f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:35076/0 (socket says 172.21.15.67:35076) 2024-03-29T11:27:13.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.552+0000 7f6c7effd640 1 -- 172.21.15.67:0/597315547 learned_addr learned my addr 172.21.15.67:0/597315547 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:13.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.552+0000 7f6c7e7fc640 1 --1- 172.21.15.67:0/597315547 >> v1:172.21.15.67:6790/0 conn(0x7f6c80143d10 0x7f6c801dec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:44084/0 (socket says 172.21.15.67:44084) 2024-03-29T11:27:13.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.552+0000 7f6c856f6640 10 monclient(hunting): picked mon.noname-a con 0x7f6c80070520 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:13.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.552+0000 7f6c856f6640 1 -- 172.21.15.67:0/597315547 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c80070ba0 con 0x7f6c80070520 2024-03-29T11:27:13.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c856f6640 1 -- 172.21.15.67:0/597315547 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c801dfa70 con 0x7f6c800b1f60 2024-03-29T11:27:13.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c856f6640 1 -- 172.21.15.67:0/597315547 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c80154340 con 0x7f6c80143d10 2024-03-29T11:27:13.557 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c856f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:13.558 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c856f6640 10 monclient(hunting): authenticate will time out at 1293.850094s 2024-03-29T11:27:13.558 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c677fe640 10 client.?.objecter ms_handle_connect 0x7f6c80143d10 2024-03-29T11:27:13.558 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:13.558 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c677fe640 10 client.?.objecter ms_handle_connect 0x7f6c80070520 2024-03-29T11:27:13.558 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:13.558 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c677fe640 10 client.?.objecter ms_handle_connect 0x7f6c800b1f60 2024-03-29T11:27:13.558 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:13.558 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123374048 0 0) 0x7f6c80154340 con 0x7f6c80143d10 2024-03-29T11:27:13.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:13.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:13.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:13.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c677fe640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:27:13.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.553+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c6c004eb0 con 0x7f6c80143d10 2024-03-29T11:27:13.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932486915 0 0) 0x7f6c801dfa70 con 0x7f6c800b1f60 2024-03-29T11:27:13.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:13.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:13.559 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:13.560 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:27:13.560 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c80154340 con 0x7f6c800b1f60 2024-03-29T11:27:13.560 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2302605614 0 0) 0x7f6c80154340 con 0x7f6c800b1f60 2024-03-29T11:27:13.560 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c801dfa70 con 0x7f6c800b1f60 2024-03-29T11:27:13.560 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6c70001530 con 0x7f6c800b1f60 2024-03-29T11:27:13.560 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:13.560 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:13.560 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): dump: 2024-03-29T11:27:13.560 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:13.561 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:13.561 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:13.561 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:13.561 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:13.561 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:13.561 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:13.561 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:13.561 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:13.562 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:13.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:13.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635667414 0 0) 0x7f6c80070ba0 con 0x7f6c80070520 2024-03-29T11:27:13.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:13.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:13.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:13.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:27:13.562 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c80154340 con 0x7f6c80070520 2024-03-29T11:27:13.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3656518829 0 0) 0x7f6c6c004eb0 con 0x7f6c80143d10 2024-03-29T11:27:13.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c80070ba0 con 0x7f6c80143d10 2024-03-29T11:27:13.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6c74001a10 con 0x7f6c80143d10 2024-03-29T11:27:13.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:13.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:13.563 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.554+0000 7f6c677fe640 10 monclient(hunting): dump: 2024-03-29T11:27:13.563 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:13.563 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:13.563 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:13.564 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:13.564 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:13.564 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:13.564 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:13.564 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:13.564 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:13.565 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:13.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2820049959 0 0) 0x7f6c80154340 con 0x7f6c80070520 2024-03-29T11:27:13.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c6c004eb0 con 0x7f6c80070520 2024-03-29T11:27:13.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6c78001b90 con 0x7f6c80070520 2024-03-29T11:27:13.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:13.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:13.565 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 10 monclient(hunting): dump: 2024-03-29T11:27:13.565 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:13.566 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:13.566 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:13.566 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:13.566 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:13.566 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:13.566 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:13.566 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:13.567 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:13.567 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:13.567 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1271823010 0 0) 0x7f6c80070ba0 con 0x7f6c80143d10 2024-03-29T11:27:13.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 >> v1:172.21.15.82:6789/0 conn(0x7f6c800b1f60 legacy=0x7f6c80154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:13.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 >> v1:172.21.15.67:6789/0 conn(0x7f6c80070520 legacy=0x7f6c801df360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:13.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:13.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 1 monclient: found mon.c 2024-03-29T11:27:13.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:13.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:13.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c80154620 con 0x7f6c80143d10 2024-03-29T11:27:13.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:13.570 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:13.571 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:13.571 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c856f6640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:27:13.571 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c856f6640 10 monclient: _renew_subs 2024-03-29T11:27:13.571 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c856f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:13.571 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c856f6640 1 -- 172.21.15.67:0/597315547 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c80154900 con 0x7f6c80143d10 2024-03-29T11:27:13.571 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c856f6640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:13.571 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c856f6640 10 monclient: _renew_subs 2024-03-29T11:27:13.571 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c856f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:13.571 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c856f6640 1 -- 172.21.15.67:0/597315547 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c80071380 con 0x7f6c80143d10 2024-03-29T11:27:13.571 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.555+0000 7f6c856f6640 1 librados: init done 2024-03-29T11:27:13.571 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.556+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c740035f0 con 0x7f6c80143d10 2024-03-29T11:27:13.571 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.556+0000 7f6c677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:13.572 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.556+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6c740015f0 con 0x7f6c80143d10 2024-03-29T11:27:13.572 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.556+0000 7f6c7d7fa640 4 set_mon_vals no callback set 2024-03-29T11:27:13.572 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.556+0000 7f6c677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:13.572 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.556+0000 7f6c677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:13.572 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.556+0000 7f6c677fe640 10 monclient: dump: 2024-03-29T11:27:13.572 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:13.572 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:13.572 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:13.572 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:13.572 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:13.573 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:13.573 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:13.573 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:13.573 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:13.573 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:13.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.558+0000 7f6c856f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:13.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.558+0000 7f6c856f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:13.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.558+0000 7f6c856f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:13.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.558+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f6c74003980 con 0x7f6c80143d10 2024-03-29T11:27:13.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.558+0000 7f6c856f6640 1 -- 172.21.15.67:0/597315547 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c60005080 con 0x7f6c80143d10 2024-03-29T11:27:13.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.559+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f6c74084930 con 0x7f6c80143d10 2024-03-29T11:27:13.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.559+0000 7f6c677fe640 10 client.4547.objecter ms_dispatch 0x7f6c80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:13.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.559+0000 7f6c677fe640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:13.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.559+0000 7f6c677fe640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:13.574 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.559+0000 7f6c677fe640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T11:27:13.574 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.559+0000 7f6c7effd640 10 monclient: get_auth_request con 0x7f6c6c06e110 auth_method 0 2024-03-29T11:27:13.574 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.560+0000 7f6c677fe640 10 client.4547.objecter ms_handle_connect 0x7f6c6c06e110 2024-03-29T11:27:13.574 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.565+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c74050210 con 0x7f6c80143d10 2024-03-29T11:27:13.574 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.566+0000 7f6c677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:13.574 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.566+0000 7f6c677fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:13.933 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.944+0000 7f6c856f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:13.933 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.944+0000 7f6c856f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:13.933 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.944+0000 7f6c856f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:13.933 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.944+0000 7f6c856f6640 1 -- 172.21.15.67:0/597315547 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c60005330 con 0x7f6c80143d10 2024-03-29T11:27:13.935 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.947+0000 7f6c677fe640 1 -- 172.21.15.67:0/597315547 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f6c74052140 con 0x7f6c80143d10 2024-03-29T11:27:13.935 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.947+0000 7f6c677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:13.936 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.947+0000 7f6c677fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:13.936 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:27:13.936 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:13.939 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.950+0000 7f6c856f6640 10 librados: watch_flush enter 2024-03-29T11:27:13.939 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.950+0000 7f6c856f6640 10 librados: watch_flush exit 2024-03-29T11:27:13.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.950+0000 7f6c856f6640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T11:27:13.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.950+0000 7f6c856f6640 10 client.4547.objecter successfully canceled tick 2024-03-29T11:27:13.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.950+0000 7f6c856f6640 1 -- 172.21.15.67:0/597315547 >> v1:172.21.15.67:6816/404787197 conn(0x7f6c6c06e110 legacy=0x7f6c6c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:13.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.950+0000 7f6c856f6640 10 monclient: shutdown 2024-03-29T11:27:13.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.950+0000 7f6c856f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:13.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.950+0000 7f6c856f6640 1 -- 172.21.15.67:0/597315547 >> v1:172.21.15.67:6790/0 conn(0x7f6c80143d10 legacy=0x7f6c801dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:13.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.950+0000 7f6c856f6640 1 -- 172.21.15.67:0/597315547 shutdown_connections 2024-03-29T11:27:13.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.950+0000 7f6c856f6640 1 -- 172.21.15.67:0/597315547 >> 172.21.15.67:0/597315547 conn(0x7f6c80066a00 msgr2=0x7f6c80146960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:13.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.950+0000 7f6c856f6640 1 -- 172.21.15.67:0/597315547 shutdown_connections 2024-03-29T11:27:13.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.950+0000 7f6c856f6640 1 -- 172.21.15.67:0/597315547 wait complete. 2024-03-29T11:27:13.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:13.951+0000 7f6c856f6640 1 librados: shutdown 2024-03-29T11:27:13.954 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:20.762 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:20.910 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.922+0000 7fd74274d640 10 monclient: get_monmap_and_config 2024-03-29T11:27:20.910 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.922+0000 7fd74274d640 10 monclient: build_initial_monmap 2024-03-29T11:27:20.910 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.922+0000 7fd74274d640 10 monclient: monmap: 2024-03-29T11:27:20.910 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:20.911 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:20.911 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:27:20.923596+0000 2024-03-29T11:27:20.911 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:27:20.923596+0000 2024-03-29T11:27:20.911 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:20.911 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:20.911 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:20.911 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:27:20.911 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:27:20.911 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:20.912 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.924+0000 7fd74274d640 1 Processor -- start 2024-03-29T11:27:20.912 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.924+0000 7fd74274d640 1 -- start start 2024-03-29T11:27:20.912 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.924+0000 7fd74274d640 10 monclient: init 2024-03-29T11:27:20.917 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.924+0000 7fd74274d640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:20.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.924+0000 7fd74274d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:20.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.925+0000 7fd74274d640 10 monclient(hunting): picked mon.noname-c con 0x7fd73c0b0860 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:20.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.925+0000 7fd74274d640 10 monclient(hunting): picked mon.noname-b con 0x7fd73c13ef30 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:20.918 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.925+0000 7fd74274d640 10 monclient(hunting): picked mon.noname-a con 0x7fd73c0af8e0 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:20.919 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.925+0000 7fd74274d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73c0b0c40 con 0x7fd73c0af8e0 2024-03-29T11:27:20.919 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.925+0000 7fd74274d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73c0aed10 con 0x7fd73c0b0860 2024-03-29T11:27:20.919 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.925+0000 7fd74274d640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73c0afd50 con 0x7fd73c13ef30 2024-03-29T11:27:20.919 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.925+0000 7fd74274d640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:20.920 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.925+0000 7fd74274d640 10 monclient(hunting): authenticate will time out at 1301.221009s 2024-03-29T11:27:20.920 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.926+0000 7fd73bfff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fd73c0b0860 0x7fd73c13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:35718/0 (socket says 172.21.15.67:35718) 2024-03-29T11:27:20.920 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.926+0000 7fd73bfff640 1 -- 172.21.15.67:0/1539523513 learned_addr learned my addr 172.21.15.67:0/1539523513 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:20.920 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.927+0000 7fd73affd640 1 -- 172.21.15.67:0/1539523513 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891406691 0 0) 0x7fd73c0aed10 con 0x7fd73c0b0860 2024-03-29T11:27:20.921 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.927+0000 7fd73affd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:20.921 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.927+0000 7fd73affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:20.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.927+0000 7fd73affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:20.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.927+0000 7fd73affd640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:27:20.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.927+0000 7fd73affd640 1 -- 172.21.15.67:0/1539523513 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd724003520 con 0x7fd73c0b0860 2024-03-29T11:27:20.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.927+0000 7fd73affd640 1 -- 172.21.15.67:0/1539523513 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 878516921 0 0) 0x7fd724003520 con 0x7fd73c0b0860 2024-03-29T11:27:20.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.927+0000 7fd73affd640 1 -- 172.21.15.67:0/1539523513 >> v1:172.21.15.67:6790/0 conn(0x7fd73c13ef30 legacy=0x7fd73c0af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:20.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.928+0000 7fd73affd640 1 -- 172.21.15.67:0/1539523513 >> v1:172.21.15.67:6789/0 conn(0x7fd73c0af8e0 legacy=0x7fd73c149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:20.922 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd73affd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:20.931 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd73affd640 1 monclient: found mon.noname-c 2024-03-29T11:27:20.931 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd73affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:20.931 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd73affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:27:20.931 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd73affd640 1 -- 172.21.15.67:0/1539523513 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c0ab1b0 con 0x7fd73c0b0860 2024-03-29T11:27:20.931 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd73affd640 10 monclient: _finish_auth 0 2024-03-29T11:27:20.931 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd73affd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:20.931 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd73affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:20.931 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd74274d640 5 monclient: authenticate success, global_id 4456 2024-03-29T11:27:20.931 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd74274d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:20.931 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd73affd640 1 -- 172.21.15.67:0/1539523513 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd734001a10 con 0x7fd73c0b0860 2024-03-29T11:27:20.931 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd73affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:20.931 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd73affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:20.931 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd73affd640 10 monclient: dump: 2024-03-29T11:27:20.932 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:20.932 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:20.932 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:20.932 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:20.932 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:20.932 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:20.932 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:20.932 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:20.932 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:20.932 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:20.932 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd74274d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:20.932 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd73affd640 1 -- 172.21.15.67:0/1539523513 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd734001ca0 con 0x7fd73c0b0860 2024-03-29T11:27:20.932 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd73affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:20.932 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd74274d640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:20.933 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.929+0000 7fd74274d640 4 set_mon_vals no callback set 2024-03-29T11:27:20.933 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.930+0000 7fd74274d640 10 monclient: shutdown 2024-03-29T11:27:20.933 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.930+0000 7fd74274d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:20.933 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.930+0000 7fd74274d640 1 -- 172.21.15.67:0/1539523513 >> v1:172.21.15.82:6789/0 conn(0x7fd73c0b0860 legacy=0x7fd73c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:20.933 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.930+0000 7fd74274d640 1 -- 172.21.15.67:0/1539523513 shutdown_connections 2024-03-29T11:27:20.933 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.930+0000 7fd74274d640 1 -- 172.21.15.67:0/1539523513 >> 172.21.15.67:0/1539523513 conn(0x7fd73c06b660 msgr2=0x7fd73c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:20.934 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.931+0000 7fd74274d640 1 -- 172.21.15.67:0/1539523513 shutdown_connections 2024-03-29T11:27:20.934 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.931+0000 7fd74274d640 1 -- 172.21.15.67:0/1539523513 wait complete. 2024-03-29T11:27:20.934 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.931+0000 7fd74274d640 10 monclient: build_initial_monmap 2024-03-29T11:27:20.934 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.932+0000 7fd74274d640 10 monclient: monmap: 2024-03-29T11:27:20.934 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:20.934 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:20.934 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:27:20.934 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:27:20.934 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:20.934 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:20.935 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:20.935 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:27:20.935 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:27:20.935 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:20.935 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.932+0000 7fd74274d640 1 librados: starting msgr at 2024-03-29T11:27:20.935 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.932+0000 7fd74274d640 1 librados: starting objecter 2024-03-29T11:27:20.935 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.933+0000 7fd74274d640 1 Processor -- start 2024-03-29T11:27:20.935 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.933+0000 7fd74274d640 1 -- start start 2024-03-29T11:27:20.935 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.933+0000 7fd74274d640 1 librados: setting wanted keys 2024-03-29T11:27:20.935 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.933+0000 7fd74274d640 1 librados: calling monclient init 2024-03-29T11:27:20.935 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.933+0000 7fd74274d640 10 monclient: init 2024-03-29T11:27:20.935 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.934+0000 7fd74274d640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:20.935 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.934+0000 7fd74274d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:20.935 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.934+0000 7fd74274d640 10 monclient(hunting): picked mon.noname-c con 0x7fd73c0af8e0 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:20.936 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.934+0000 7fd74274d640 10 monclient(hunting): picked mon.noname-b con 0x7fd73c0b0860 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:20.936 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.934+0000 7fd74274d640 10 monclient(hunting): picked mon.noname-a con 0x7fd73c13ef30 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:20.936 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.934+0000 7fd74274d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73c136d30 con 0x7fd73c13ef30 2024-03-29T11:27:20.936 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.934+0000 7fd74274d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73c132690 con 0x7fd73c0b0860 2024-03-29T11:27:20.936 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.934+0000 7fd74274d640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73c071080 con 0x7fd73c0af8e0 2024-03-29T11:27:20.936 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.934+0000 7fd74274d640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:20.936 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.934+0000 7fd74274d640 10 monclient(hunting): authenticate will time out at 1301.230798s 2024-03-29T11:27:20.936 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.936+0000 7fd73b7fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fd73c0b0860 0x7fd73c132bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:36332/0 (socket says 172.21.15.67:36332) 2024-03-29T11:27:20.936 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.936+0000 7fd73b7fe640 1 -- 172.21.15.67:0/1622500951 learned_addr learned my addr 172.21.15.67:0/1622500951 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:20.936 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.936+0000 7fd738ff9640 10 client.?.objecter ms_handle_connect 0x7fd73c0b0860 2024-03-29T11:27:20.936 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.936+0000 7fd738ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:20.936 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.937+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491647353 0 0) 0x7fd73c132690 con 0x7fd73c0b0860 2024-03-29T11:27:20.937 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.937+0000 7fd738ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:20.937 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.937+0000 7fd738ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:20.937 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.937+0000 7fd738ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:20.937 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.937+0000 7fd738ff9640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:27:20.937 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.937+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd728004eb0 con 0x7fd73c0b0860 2024-03-29T11:27:20.937 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.937+0000 7fd738ff9640 10 client.?.objecter ms_handle_connect 0x7fd73c0af8e0 2024-03-29T11:27:20.937 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:20.937 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2354323886 0 0) 0x7fd728004eb0 con 0x7fd73c0b0860 2024-03-29T11:27:20.937 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd73c132690 con 0x7fd73c0b0860 2024-03-29T11:27:20.937 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910177021 0 0) 0x7fd73c071080 con 0x7fd73c0af8e0 2024-03-29T11:27:20.937 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:20.937 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:20.937 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:20.937 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:27:20.938 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd728004eb0 con 0x7fd73c0af8e0 2024-03-29T11:27:20.938 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd730003fe0 con 0x7fd73c0b0860 2024-03-29T11:27:20.938 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:20.938 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:20.938 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:20.938 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:20.938 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:20.938 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:20.938 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:20.938 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:20.938 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:20.938 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:20.938 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:20.938 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:20.939 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:20.939 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:20.939 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1245781712 0 0) 0x7fd728004eb0 con 0x7fd73c0af8e0 2024-03-29T11:27:20.939 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd73c071080 con 0x7fd73c0af8e0 2024-03-29T11:27:20.939 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd734004c90 con 0x7fd73c0af8e0 2024-03-29T11:27:20.939 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:20.939 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:20.939 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.938+0000 7fd738ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:20.939 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:20.939 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:20.939 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:20.939 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:20.939 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:20.939 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:20.940 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:20.940 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:20.940 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:20.940 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:20.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.939+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1791790119 0 0) 0x7fd73c071080 con 0x7fd73c0af8e0 2024-03-29T11:27:20.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.939+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 >> v1:172.21.15.82:6789/0 conn(0x7fd73c0b0860 legacy=0x7fd73c132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:20.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.939+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 >> v1:172.21.15.67:6789/0 conn(0x7fd73c13ef30 legacy=0x7fd73c1e7060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:20.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd738ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:20.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd738ff9640 1 monclient: found mon.c 2024-03-29T11:27:20.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd738ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:20.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd738ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:20.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c071210 con 0x7fd73c0af8e0 2024-03-29T11:27:20.940 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd738ff9640 10 monclient: _finish_auth 0 2024-03-29T11:27:20.944 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd738ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:20.945 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd738ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:20.945 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd74274d640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:27:20.945 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd74274d640 10 monclient: _renew_subs 2024-03-29T11:27:20.945 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd74274d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:20.945 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd74274d640 1 -- 172.21.15.67:0/1622500951 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73c130de0 con 0x7fd73c0af8e0 2024-03-29T11:27:20.945 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd74274d640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:20.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd74274d640 10 monclient: _renew_subs 2024-03-29T11:27:20.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd74274d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:20.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.941+0000 7fd74274d640 1 -- 172.21.15.67:0/1622500951 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd73c1320b0 con 0x7fd73c0af8e0 2024-03-29T11:27:20.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.942+0000 7fd74274d640 1 librados: init done 2024-03-29T11:27:20.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.943+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd734001e90 con 0x7fd73c0af8e0 2024-03-29T11:27:20.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.943+0000 7fd738ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:20.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.943+0000 7fd73a7fc640 4 set_mon_vals no callback set 2024-03-29T11:27:20.946 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.943+0000 7fd74274d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:20.947 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.944+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fd734005240 con 0x7fd73c0af8e0 2024-03-29T11:27:20.947 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.944+0000 7fd738ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:20.947 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.944+0000 7fd738ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:20.947 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.944+0000 7fd738ff9640 10 monclient: dump: 2024-03-29T11:27:20.947 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:20.949 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:20.950 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:20.950 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:20.950 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:20.950 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:20.950 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:20.950 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:20.951 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:20.951 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:20.951 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.944+0000 7fd74274d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:20.951 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.944+0000 7fd74274d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:20.951 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.945+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fd734005460 con 0x7fd73c0af8e0 2024-03-29T11:27:20.951 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.944+0000 7fd74274d640 1 -- 172.21.15.67:0/1622500951 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd71c005080 con 0x7fd73c0af8e0 2024-03-29T11:27:20.951 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.946+0000 7fd73b7fe640 10 monclient: get_auth_request con 0x7fd72806dfa0 auth_method 0 2024-03-29T11:27:20.951 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.947+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fd734084f00 con 0x7fd73c0af8e0 2024-03-29T11:27:20.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.947+0000 7fd738ff9640 10 client.4550.objecter ms_dispatch 0x7fd73c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:20.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.947+0000 7fd738ff9640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:20.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.947+0000 7fd738ff9640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:20.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.947+0000 7fd738ff9640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-29T11:27:20.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.948+0000 7fd738ff9640 10 client.4550.objecter ms_handle_connect 0x7fd72806dfa0 2024-03-29T11:27:20.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.953+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd7340507e0 con 0x7fd73c0af8e0 2024-03-29T11:27:20.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.953+0000 7fd738ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:20.952 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:20.953+0000 7fd738ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:21.286 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.299+0000 7fd74274d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.287 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.299+0000 7fd74274d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.287 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.299+0000 7fd74274d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:21.287 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.299+0000 7fd74274d640 1 -- 172.21.15.67:0/1622500951 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd71c005330 con 0x7fd73c0af8e0 2024-03-29T11:27:21.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.303+0000 7fd738ff9640 1 -- 172.21.15.67:0/1622500951 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fd734052710 con 0x7fd73c0af8e0 2024-03-29T11:27:21.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.303+0000 7fd738ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.303+0000 7fd738ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:21.291 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:27:21.291 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:21.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.307+0000 7fd74274d640 10 librados: watch_flush enter 2024-03-29T11:27:21.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.307+0000 7fd74274d640 10 librados: watch_flush exit 2024-03-29T11:27:21.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.307+0000 7fd74274d640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-29T11:27:21.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.307+0000 7fd74274d640 10 client.4550.objecter successfully canceled tick 2024-03-29T11:27:21.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.307+0000 7fd74274d640 1 -- 172.21.15.67:0/1622500951 >> v1:172.21.15.67:6816/404787197 conn(0x7fd72806dfa0 legacy=0x7fd728070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.307+0000 7fd74274d640 10 monclient: shutdown 2024-03-29T11:27:21.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.307+0000 7fd74274d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:21.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.307+0000 7fd74274d640 1 -- 172.21.15.67:0/1622500951 >> v1:172.21.15.67:6790/0 conn(0x7fd73c0af8e0 legacy=0x7fd73c1306d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.307+0000 7fd74274d640 1 -- 172.21.15.67:0/1622500951 shutdown_connections 2024-03-29T11:27:21.295 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.307+0000 7fd74274d640 1 -- 172.21.15.67:0/1622500951 >> 172.21.15.67:0/1622500951 conn(0x7fd73c06b660 msgr2=0x7fd73c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:21.296 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.307+0000 7fd74274d640 1 -- 172.21.15.67:0/1622500951 shutdown_connections 2024-03-29T11:27:21.296 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.307+0000 7fd74274d640 1 -- 172.21.15.67:0/1622500951 wait complete. 2024-03-29T11:27:21.296 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:21.307+0000 7fd74274d640 1 librados: shutdown 2024-03-29T11:27:21.314 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:28.125 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:28.285 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.297+0000 7f2efa1e6640 10 monclient: get_monmap_and_config 2024-03-29T11:27:28.285 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.297+0000 7f2efa1e6640 10 monclient: build_initial_monmap 2024-03-29T11:27:28.285 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.297+0000 7f2efa1e6640 10 monclient: monmap: 2024-03-29T11:27:28.285 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:28.285 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:28.286 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:27:28.298156+0000 2024-03-29T11:27:28.286 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:27:28.298156+0000 2024-03-29T11:27:28.286 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:28.286 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:28.286 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:28.286 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:27:28.286 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:27:28.286 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:28.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.300+0000 7f2efa1e6640 1 Processor -- start 2024-03-29T11:27:28.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.300+0000 7f2efa1e6640 1 -- start start 2024-03-29T11:27:28.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.300+0000 7f2efa1e6640 10 monclient: init 2024-03-29T11:27:28.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.301+0000 7f2efa1e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:28.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.301+0000 7f2efa1e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:28.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.301+0000 7f2efa1e6640 10 monclient(hunting): picked mon.noname-a con 0x7f2ef40b0860 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:28.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.301+0000 7f2efa1e6640 10 monclient(hunting): picked mon.noname-c con 0x7f2ef413ef30 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:28.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.301+0000 7f2efa1e6640 10 monclient(hunting): picked mon.noname-b con 0x7f2ef40af8e0 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:28.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.301+0000 7f2efa1e6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef40b0c40 con 0x7f2ef40b0860 2024-03-29T11:27:28.289 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.301+0000 7f2efa1e6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef40aed10 con 0x7f2ef413ef30 2024-03-29T11:27:28.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.301+0000 7f2efa1e6640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef40afd50 con 0x7f2ef40af8e0 2024-03-29T11:27:28.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.301+0000 7f2efa1e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:28.290 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.301+0000 7f2efa1e6640 10 monclient(hunting): authenticate will time out at 1308.597346s 2024-03-29T11:27:28.296 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.308+0000 7f2ef2ffd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f2ef413ef30 0x7f2ef40af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:34097/0 (socket says 172.21.15.67:34097) 2024-03-29T11:27:28.296 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.308+0000 7f2ef2ffd640 1 -- 172.21.15.67:0/2900255800 learned_addr learned my addr 172.21.15.67:0/2900255800 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:28.297 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.309+0000 7f2ef27fc640 1 -- 172.21.15.67:0/2900255800 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940198320 0 0) 0x7f2ef40aed10 con 0x7f2ef413ef30 2024-03-29T11:27:28.297 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.309+0000 7f2ef27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.297 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.309+0000 7f2ef27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.297 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.309+0000 7f2ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.298 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.309+0000 7f2ef27fc640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:27:28.298 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.309+0000 7f2ef27fc640 1 -- 172.21.15.67:0/2900255800 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2edc003520 con 0x7f2ef413ef30 2024-03-29T11:27:28.298 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.309+0000 7f2ef27fc640 1 -- 172.21.15.67:0/2900255800 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137575619 0 0) 0x7f2ef40afd50 con 0x7f2ef40af8e0 2024-03-29T11:27:28.298 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.309+0000 7f2ef27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.298 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.309+0000 7f2ef27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.298 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.309+0000 7f2ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.298 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.309+0000 7f2ef27fc640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:27:28.298 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.309+0000 7f2ef27fc640 1 -- 172.21.15.67:0/2900255800 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef40aed10 con 0x7f2ef40af8e0 2024-03-29T11:27:28.299 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 1 -- 172.21.15.67:0/2900255800 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2048747023 0 0) 0x7f2edc003520 con 0x7f2ef413ef30 2024-03-29T11:27:28.299 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 1 -- 172.21.15.67:0/2900255800 >> v1:172.21.15.67:6790/0 conn(0x7f2ef40af8e0 legacy=0x7f2ef4149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.299 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 1 -- 172.21.15.67:0/2900255800 >> v1:172.21.15.67:6789/0 conn(0x7f2ef40b0860 legacy=0x7f2ef413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:28.299 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:28.299 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 1 monclient: found mon.noname-c 2024-03-29T11:27:28.299 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:28.299 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:27:28.299 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 1 -- 172.21.15.67:0/2900255800 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef40ab1b0 con 0x7f2ef413ef30 2024-03-29T11:27:28.299 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:28.299 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:28.300 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:28.300 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 1 -- 172.21.15.67:0/2900255800 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f2eec001690 con 0x7f2ef413ef30 2024-03-29T11:27:28.300 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.300 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:28.300 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 10 monclient: dump: 2024-03-29T11:27:28.300 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:28.300 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:28.300 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:28.300 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:28.300 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.300 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:28.300 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:28.300 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:28.300 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:28.301 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:28.301 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2efa1e6640 5 monclient: authenticate success, global_id 4462 2024-03-29T11:27:28.301 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2efa1e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:28.301 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 1 -- 172.21.15.67:0/2900255800 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2eec001870 con 0x7f2ef413ef30 2024-03-29T11:27:28.301 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:28.301 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 1 -- 172.21.15.67:0/2900255800 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f2eec001c80 con 0x7f2ef413ef30 2024-03-29T11:27:28.301 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.301 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:28.301 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2ef27fc640 10 monclient: dump: 2024-03-29T11:27:28.301 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:28.301 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:28.301 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:28.301 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:28.301 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.302 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:28.302 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:28.302 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:28.302 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:28.302 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:28.302 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2efa1e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:28.302 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.310+0000 7f2efa1e6640 4 set_mon_vals no callback set 2024-03-29T11:27:28.302 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.311+0000 7f2efa1e6640 10 monclient: shutdown 2024-03-29T11:27:28.302 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.311+0000 7f2efa1e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:28.302 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.311+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2900255800 >> v1:172.21.15.82:6789/0 conn(0x7f2ef413ef30 legacy=0x7f2ef40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.305 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.313+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2900255800 shutdown_connections 2024-03-29T11:27:28.305 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.313+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2900255800 >> 172.21.15.67:0/2900255800 conn(0x7f2ef406b660 msgr2=0x7f2ef406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:28.305 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.314+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2900255800 shutdown_connections 2024-03-29T11:27:28.305 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.314+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2900255800 wait complete. 2024-03-29T11:27:28.305 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.315+0000 7f2efa1e6640 10 monclient: build_initial_monmap 2024-03-29T11:27:28.305 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.315+0000 7f2efa1e6640 10 monclient: monmap: 2024-03-29T11:27:28.305 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:28.305 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:28.305 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:27:28.305 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:27:28.305 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:28.305 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:28.306 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:28.306 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:27:28.306 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:27:28.306 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:28.306 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.316+0000 7f2efa1e6640 1 librados: starting msgr at 2024-03-29T11:27:28.306 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.316+0000 7f2efa1e6640 1 librados: starting objecter 2024-03-29T11:27:28.306 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.316+0000 7f2efa1e6640 1 Processor -- start 2024-03-29T11:27:28.306 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.317+0000 7f2efa1e6640 1 -- start start 2024-03-29T11:27:28.306 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.317+0000 7f2efa1e6640 1 librados: setting wanted keys 2024-03-29T11:27:28.306 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.317+0000 7f2efa1e6640 1 librados: calling monclient init 2024-03-29T11:27:28.306 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.317+0000 7f2efa1e6640 10 monclient: init 2024-03-29T11:27:28.307 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.318+0000 7f2efa1e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:28.307 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.318+0000 7f2efa1e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:28.307 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.318+0000 7f2efa1e6640 10 monclient(hunting): picked mon.noname-c con 0x7f2ef40af8e0 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:28.309 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.321+0000 7f2efa1e6640 10 monclient(hunting): picked mon.noname-b con 0x7f2ef40b0860 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:28.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.324+0000 7f2ef37fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f2ef40af8e0 0x7f2ef4137d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:45253/0 (socket says 172.21.15.67:45253) 2024-03-29T11:27:28.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.324+0000 7f2ef37fe640 1 -- 172.21.15.67:0/2019059579 learned_addr learned my addr 172.21.15.67:0/2019059579 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:28.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.324+0000 7f2efa1e6640 10 monclient(hunting): picked mon.noname-a con 0x7f2ef413ef30 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:28.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.324+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2019059579 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef40b1260 con 0x7f2ef413ef30 2024-03-29T11:27:28.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.324+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2019059579 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef40b20e0 con 0x7f2ef40b0860 2024-03-29T11:27:28.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.324+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2019059579 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef4136fa0 con 0x7f2ef40af8e0 2024-03-29T11:27:28.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2efa1e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:28.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2efa1e6640 10 monclient(hunting): authenticate will time out at 1308.621091s 2024-03-29T11:27:28.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 10 client.?.objecter ms_handle_connect 0x7f2ef40af8e0 2024-03-29T11:27:28.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:28.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734527041 0 0) 0x7f2ef4136fa0 con 0x7f2ef40af8e0 2024-03-29T11:27:28.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:27:28.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee0004eb0 con 0x7f2ef40af8e0 2024-03-29T11:27:28.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3799179002 0 0) 0x7f2ee0004eb0 con 0x7f2ef40af8e0 2024-03-29T11:27:28.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ef4136fa0 con 0x7f2ef40af8e0 2024-03-29T11:27:28.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f2ee4004400 con 0x7f2ef40af8e0 2024-03-29T11:27:28.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:28.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 10 monclient(hunting): dump: 2024-03-29T11:27:28.320 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:28.320 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:28.320 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:28.320 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:28.321 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.321 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:28.321 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:28.321 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:28.321 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:28.321 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:28.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:28.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2027797291 0 0) 0x7f2ef4136fa0 con 0x7f2ef40af8e0 2024-03-29T11:27:28.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 >> v1:172.21.15.82:6789/0 conn(0x7f2ef40b0860 legacy=0x7f2ef40b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.325+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 >> v1:172.21.15.67:6789/0 conn(0x7f2ef413ef30 legacy=0x7f2ef40b31d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:28.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.326+0000 7f2edbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:28.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.326+0000 7f2edbfff640 1 monclient: found mon.c 2024-03-29T11:27:28.321 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.326+0000 7f2edbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:28.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.326+0000 7f2edbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:28.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.326+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef40b2350 con 0x7f2ef40af8e0 2024-03-29T11:27:28.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.327+0000 7f2edbfff640 10 monclient: _finish_auth 0 2024-03-29T11:27:28.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.327+0000 7f2edbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:28.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.327+0000 7f2edbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:28.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.327+0000 7f2efa1e6640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:27:28.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.327+0000 7f2efa1e6640 10 monclient: _renew_subs 2024-03-29T11:27:28.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.327+0000 7f2efa1e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:28.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.327+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2019059579 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef4071a80 con 0x7f2ef40af8e0 2024-03-29T11:27:28.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.327+0000 7f2efa1e6640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:28.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.327+0000 7f2efa1e6640 10 monclient: _renew_subs 2024-03-29T11:27:28.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.327+0000 7f2efa1e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.327+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2019059579 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef41386c0 con 0x7f2ef40af8e0 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.327+0000 7f2efa1e6640 1 librados: init done 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.327+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ee40047e0 con 0x7f2ef40af8e0 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.328+0000 7f2edbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.328+0000 7f2ef1ffb640 4 set_mon_vals no callback set 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.329+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f2ee4001500 con 0x7f2ef40af8e0 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.329+0000 7f2edbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.329+0000 7f2efa1e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.329+0000 7f2edbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.329+0000 7f2edbfff640 10 monclient: dump: 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:28.323 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.324 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:28.324 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:28.324 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:28.324 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:28.324 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:28.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.329+0000 7f2efa1e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.329+0000 7f2efa1e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:28.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.329+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2019059579 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed4005080 con 0x7f2ef40af8e0 2024-03-29T11:27:28.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.329+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f2ee4001660 con 0x7f2ef40af8e0 2024-03-29T11:27:28.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.330+0000 7f2ef2ffd640 10 monclient: get_auth_request con 0x7f2ee006dbd0 auth_method 0 2024-03-29T11:27:28.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.331+0000 7f2edbfff640 10 client.4556.objecter ms_handle_connect 0x7f2ee006dbd0 2024-03-29T11:27:28.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.331+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f2ee4084e80 con 0x7f2ef40af8e0 2024-03-29T11:27:28.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.331+0000 7f2edbfff640 10 client.4556.objecter ms_dispatch 0x7f2ef4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:28.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.332+0000 7f2edbfff640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:28.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.332+0000 7f2edbfff640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:28.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.332+0000 7f2edbfff640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T11:27:28.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.339+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ee4050590 con 0x7f2ef40af8e0 2024-03-29T11:27:28.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.339+0000 7f2edbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.339+0000 7f2edbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:28.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.681+0000 7f2efa1e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:28.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.681+0000 7f2efa1e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:28.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.681+0000 7f2efa1e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:28.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.681+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2019059579 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ed4005330 con 0x7f2ef40af8e0 2024-03-29T11:27:28.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.682+0000 7f2edbfff640 1 -- 172.21.15.67:0/2019059579 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f2ee4052690 con 0x7f2ef40af8e0 2024-03-29T11:27:28.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.682+0000 7f2edbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:28.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.682+0000 7f2edbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:28.670 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:27:28.670 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:28.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.685+0000 7f2efa1e6640 10 librados: watch_flush enter 2024-03-29T11:27:28.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.686+0000 7f2efa1e6640 10 librados: watch_flush exit 2024-03-29T11:27:28.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.686+0000 7f2efa1e6640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T11:27:28.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.686+0000 7f2efa1e6640 10 client.4556.objecter successfully canceled tick 2024-03-29T11:27:28.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.686+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2019059579 >> v1:172.21.15.67:6816/404787197 conn(0x7f2ee006dbd0 legacy=0x7f2ee0070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.686+0000 7f2efa1e6640 10 monclient: shutdown 2024-03-29T11:27:28.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.686+0000 7f2efa1e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:28.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.686+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2019059579 >> v1:172.21.15.67:6790/0 conn(0x7f2ef40af8e0 legacy=0x7f2ef4137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.686+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2019059579 shutdown_connections 2024-03-29T11:27:28.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.686+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2019059579 >> 172.21.15.67:0/2019059579 conn(0x7f2ef406b660 msgr2=0x7f2ef41c91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:28.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.687+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2019059579 shutdown_connections 2024-03-29T11:27:28.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.687+0000 7f2efa1e6640 1 -- 172.21.15.67:0/2019059579 wait complete. 2024-03-29T11:27:28.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:28.687+0000 7f2efa1e6640 1 librados: shutdown 2024-03-29T11:27:28.692 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:35.502 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:35.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.662+0000 7f22ff82a640 10 monclient: get_monmap_and_config 2024-03-29T11:27:35.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.662+0000 7f22ff82a640 10 monclient: build_initial_monmap 2024-03-29T11:27:35.651 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.662+0000 7f22ff82a640 10 monclient: monmap: 2024-03-29T11:27:35.651 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:35.651 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:35.651 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:27:35.663041+0000 2024-03-29T11:27:35.651 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:27:35.663041+0000 2024-03-29T11:27:35.651 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:35.652 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:35.652 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:35.652 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:27:35.652 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:27:35.652 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:35.653 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.663+0000 7f22ff82a640 1 Processor -- start 2024-03-29T11:27:35.653 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.663+0000 7f22ff82a640 1 -- start start 2024-03-29T11:27:35.653 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.663+0000 7f22ff82a640 10 monclient: init 2024-03-29T11:27:35.653 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.664+0000 7f22ff82a640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:35.653 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.664+0000 7f22ff82a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:35.654 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.666+0000 7f22ff82a640 10 monclient(hunting): picked mon.noname-a con 0x7f22f806d9d0 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:35.654 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.666+0000 7f22ff82a640 10 monclient(hunting): picked mon.noname-c con 0x7f22f813ef30 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:35.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.666+0000 7f22ff82a640 10 monclient(hunting): picked mon.noname-b con 0x7f22f80b07d0 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:35.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.666+0000 7f22ff82a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f80aed10 con 0x7f22f806d9d0 2024-03-29T11:27:35.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.666+0000 7f22ff82a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f80b0c40 con 0x7f22f813ef30 2024-03-29T11:27:35.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.666+0000 7f22ff82a640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f80ab1b0 con 0x7f22f80b07d0 2024-03-29T11:27:35.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.666+0000 7f22ff82a640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:35.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.666+0000 7f22ff82a640 10 monclient(hunting): authenticate will time out at 1315.962155s 2024-03-29T11:27:35.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.666+0000 7f22fddc7640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f22f80b07d0 0x7f22f814ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:55154/0 (socket says 172.21.15.67:55154) 2024-03-29T11:27:35.655 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.666+0000 7f22fddc7640 1 -- 172.21.15.67:0/103098474 learned_addr learned my addr 172.21.15.67:0/103098474 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:35.656 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.667+0000 7f22effff640 1 -- 172.21.15.67:0/103098474 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932446081 0 0) 0x7f22f80ab1b0 con 0x7f22f80b07d0 2024-03-29T11:27:35.656 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.667+0000 7f22effff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:35.656 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.667+0000 7f22effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:35.656 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.667+0000 7f22effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:35.656 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.667+0000 7f22effff640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:27:35.656 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.667+0000 7f22effff640 1 -- 172.21.15.67:0/103098474 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e4003520 con 0x7f22f80b07d0 2024-03-29T11:27:35.656 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.667+0000 7f22effff640 1 -- 172.21.15.67:0/103098474 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375570248 0 0) 0x7f22f80aed10 con 0x7f22f806d9d0 2024-03-29T11:27:35.656 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.667+0000 7f22effff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:35.657 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.667+0000 7f22effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:35.657 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.667+0000 7f22effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:35.657 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.667+0000 7f22effff640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:27:35.657 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.667+0000 7f22effff640 1 -- 172.21.15.67:0/103098474 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f80ab1b0 con 0x7f22f806d9d0 2024-03-29T11:27:35.657 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.668+0000 7f22effff640 1 -- 172.21.15.67:0/103098474 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3269592091 0 0) 0x7f22f80ab1b0 con 0x7f22f806d9d0 2024-03-29T11:27:35.657 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.668+0000 7f22effff640 1 -- 172.21.15.67:0/103098474 >> v1:172.21.15.67:6790/0 conn(0x7f22f80b07d0 legacy=0x7f22f814ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:35.657 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.668+0000 7f22effff640 1 -- 172.21.15.67:0/103098474 >> v1:172.21.15.82:6789/0 conn(0x7f22f813ef30 legacy=0x7f22f80af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:35.662 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:35.662 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 1 monclient: found mon.noname-a 2024-03-29T11:27:35.662 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:35.662 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:27:35.663 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 1 -- 172.21.15.67:0/103098474 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f80ab490 con 0x7f22f806d9d0 2024-03-29T11:27:35.663 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 10 monclient: _finish_auth 0 2024-03-29T11:27:35.663 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:35.663 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:35.663 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 1 -- 172.21.15.67:0/103098474 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f22e80015f0 con 0x7f22f806d9d0 2024-03-29T11:27:35.663 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:35.663 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:35.663 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 10 monclient: dump: 2024-03-29T11:27:35.663 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:35.663 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:35.663 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:35.663 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:35.663 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:35.664 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:35.664 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:35.664 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:35.664 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:35.664 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:35.664 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22ff82a640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:27:35.664 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22ff82a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:35.664 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 1 -- 172.21.15.67:0/103098474 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22e8001750 con 0x7f22f806d9d0 2024-03-29T11:27:35.664 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:35.664 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 1 -- 172.21.15.67:0/103098474 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f22e8001ba0 con 0x7f22f806d9d0 2024-03-29T11:27:35.664 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:35.665 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:35.665 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22effff640 10 monclient: dump: 2024-03-29T11:27:35.665 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:35.665 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:35.665 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:35.665 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:35.665 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:35.666 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:35.666 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:35.666 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:35.666 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:35.666 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:35.666 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22ff82a640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:35.666 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.670+0000 7f22ff82a640 4 set_mon_vals no callback set 2024-03-29T11:27:35.666 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.671+0000 7f22ff82a640 10 monclient: shutdown 2024-03-29T11:27:35.666 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.671+0000 7f22ff82a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:35.666 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.671+0000 7f22ff82a640 1 -- 172.21.15.67:0/103098474 >> v1:172.21.15.67:6789/0 conn(0x7f22f806d9d0 legacy=0x7f22f813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:35.666 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.672+0000 7f22ff82a640 1 -- 172.21.15.67:0/103098474 shutdown_connections 2024-03-29T11:27:35.667 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.672+0000 7f22ff82a640 1 -- 172.21.15.67:0/103098474 >> 172.21.15.67:0/103098474 conn(0x7f22f806b660 msgr2=0x7f22f806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:35.667 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.672+0000 7f22ff82a640 1 -- 172.21.15.67:0/103098474 shutdown_connections 2024-03-29T11:27:35.667 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.673+0000 7f22ff82a640 1 -- 172.21.15.67:0/103098474 wait complete. 2024-03-29T11:27:35.667 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.673+0000 7f22ff82a640 10 monclient: build_initial_monmap 2024-03-29T11:27:35.667 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.673+0000 7f22ff82a640 10 monclient: monmap: 2024-03-29T11:27:35.667 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:35.667 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:35.667 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:27:35.667 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:27:35.667 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:35.667 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:35.667 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:35.668 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:27:35.668 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:27:35.668 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:35.668 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.674+0000 7f22ff82a640 1 librados: starting msgr at 2024-03-29T11:27:35.668 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.674+0000 7f22ff82a640 1 librados: starting objecter 2024-03-29T11:27:35.668 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.674+0000 7f22ff82a640 1 Processor -- start 2024-03-29T11:27:35.668 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.674+0000 7f22ff82a640 1 -- start start 2024-03-29T11:27:35.668 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.674+0000 7f22ff82a640 1 librados: setting wanted keys 2024-03-29T11:27:35.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.674+0000 7f22ff82a640 1 librados: calling monclient init 2024-03-29T11:27:35.669 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.674+0000 7f22ff82a640 10 monclient: init 2024-03-29T11:27:35.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.675+0000 7f22ff82a640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:35.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.675+0000 7f22ff82a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:35.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.675+0000 7f22ff82a640 10 monclient(hunting): picked mon.noname-b con 0x7f22f806d9d0 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:35.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.676+0000 7f22ff82a640 10 monclient(hunting): picked mon.noname-c con 0x7f22f80b07d0 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:35.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.676+0000 7f22ff82a640 10 monclient(hunting): picked mon.noname-a con 0x7f22f813ef30 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:35.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.676+0000 7f22ff82a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f806f6b0 con 0x7f22f813ef30 2024-03-29T11:27:35.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.676+0000 7f22ff82a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f80703e0 con 0x7f22f806d9d0 2024-03-29T11:27:35.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.676+0000 7f22ff82a640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f81db3f0 con 0x7f22f80b07d0 2024-03-29T11:27:35.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.676+0000 7f22ff82a640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:35.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.676+0000 7f22ff82a640 10 monclient(hunting): authenticate will time out at 1315.972317s 2024-03-29T11:27:35.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.678+0000 7f22fcdc5640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f22f80b07d0 0x7f22f81e4690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:59558/0 (socket says 172.21.15.67:59558) 2024-03-29T11:27:35.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.678+0000 7f22fcdc5640 1 -- 172.21.15.67:0/2380556575 learned_addr learned my addr 172.21.15.67:0/2380556575 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:35.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.678+0000 7f22edffb640 10 client.?.objecter ms_handle_connect 0x7f22f80b07d0 2024-03-29T11:27:35.674 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.678+0000 7f22edffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:35.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.678+0000 7f22edffb640 10 client.?.objecter ms_handle_connect 0x7f22f806d9d0 2024-03-29T11:27:35.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.678+0000 7f22edffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:35.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.678+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278156294 0 0) 0x7f22f80703e0 con 0x7f22f806d9d0 2024-03-29T11:27:35.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.678+0000 7f22edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:35.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.678+0000 7f22edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:35.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.678+0000 7f22edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:35.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.678+0000 7f22edffb640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:27:35.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.679+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22dc004eb0 con 0x7f22f806d9d0 2024-03-29T11:27:35.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.679+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457919783 0 0) 0x7f22f81db3f0 con 0x7f22f80b07d0 2024-03-29T11:27:35.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.679+0000 7f22edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:35.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.679+0000 7f22edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:35.675 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.679+0000 7f22edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:35.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.679+0000 7f22edffb640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:27:35.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.679+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f80703e0 con 0x7f22f80b07d0 2024-03-29T11:27:35.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.680+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1711117511 0 0) 0x7f22f80703e0 con 0x7f22f80b07d0 2024-03-29T11:27:35.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.680+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22dc005490 con 0x7f22f80b07d0 2024-03-29T11:27:35.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.680+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f22f40017f0 con 0x7f22f80b07d0 2024-03-29T11:27:35.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.680+0000 7f22edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:35.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.680+0000 7f22edffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:35.676 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.680+0000 7f22edffb640 10 monclient(hunting): dump: 2024-03-29T11:27:35.676 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:35.676 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:35.676 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:35.676 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:35.676 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:35.677 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:35.677 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:35.677 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:35.677 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:35.677 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:35.677 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.680+0000 7f22edffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:35.677 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.681+0000 7f22edffb640 10 client.?.objecter ms_handle_connect 0x7f22f813ef30 2024-03-29T11:27:35.677 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.681+0000 7f22edffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:35.677 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.681+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316549059 0 0) 0x7f22f806f6b0 con 0x7f22f813ef30 2024-03-29T11:27:35.677 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.681+0000 7f22edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:35.678 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.681+0000 7f22edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:35.678 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.681+0000 7f22edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:35.678 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.681+0000 7f22edffb640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:27:35.678 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.681+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f80703e0 con 0x7f22f813ef30 2024-03-29T11:27:35.678 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 630486366 0 0) 0x7f22dc005490 con 0x7f22f80b07d0 2024-03-29T11:27:35.678 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 >> v1:172.21.15.82:6789/0 conn(0x7f22f806d9d0 legacy=0x7f22f81dace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:35.678 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 >> v1:172.21.15.67:6789/0 conn(0x7f22f813ef30 legacy=0x7f22f81efbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:35.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22edffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:35.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22edffb640 1 monclient: found mon.c 2024-03-29T11:27:35.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:35.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22edffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:35.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f81db580 con 0x7f22f80b07d0 2024-03-29T11:27:35.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22edffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:35.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22edffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:35.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:35.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22ff82a640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:27:35.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22ff82a640 10 monclient: _renew_subs 2024-03-29T11:27:35.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22ff82a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:35.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.682+0000 7f22ff82a640 1 -- 172.21.15.67:0/2380556575 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f80718f0 con 0x7f22f80b07d0 2024-03-29T11:27:35.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.683+0000 7f22ff82a640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:35.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.683+0000 7f22ff82a640 10 monclient: _renew_subs 2024-03-29T11:27:35.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.683+0000 7f22ff82a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:35.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.683+0000 7f22ff82a640 1 -- 172.21.15.67:0/2380556575 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f806f2c0 con 0x7f22f80b07d0 2024-03-29T11:27:35.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.683+0000 7f22ff82a640 1 librados: init done 2024-03-29T11:27:35.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.683+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22f4001d00 con 0x7f22f80b07d0 2024-03-29T11:27:35.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.683+0000 7f22edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:35.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.683+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f22f4004aa0 con 0x7f22f80b07d0 2024-03-29T11:27:35.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.683+0000 7f22edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:35.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.683+0000 7f22edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:35.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.683+0000 7f22edffb640 10 monclient: dump: 2024-03-29T11:27:35.682 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:35.682 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:35.682 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:35.682 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:35.682 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:35.682 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:35.682 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:35.682 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:35.682 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:35.682 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:35.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.683+0000 7f22ef7fe640 4 set_mon_vals no callback set 2024-03-29T11:27:35.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.684+0000 7f22ff82a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:35.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.684+0000 7f22ff82a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:35.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.684+0000 7f22ff82a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:35.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.684+0000 7f22ff82a640 1 -- 172.21.15.67:0/2380556575 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22d8005080 con 0x7f22f80b07d0 2024-03-29T11:27:35.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.686+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f22f401aa70 con 0x7f22f80b07d0 2024-03-29T11:27:35.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.687+0000 7f22fd5c6640 10 monclient: get_auth_request con 0x7f22dc06e1e0 auth_method 0 2024-03-29T11:27:35.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.687+0000 7f22edffb640 10 client.4562.objecter ms_handle_connect 0x7f22dc06e1e0 2024-03-29T11:27:35.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.687+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f22f40545a0 con 0x7f22f80b07d0 2024-03-29T11:27:35.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.687+0000 7f22edffb640 10 client.4562.objecter ms_dispatch 0x7f22f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:35.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.687+0000 7f22edffb640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:35.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.687+0000 7f22edffb640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:35.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.687+0000 7f22edffb640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-29T11:27:35.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.694+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22f4050840 con 0x7f22f80b07d0 2024-03-29T11:27:35.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.694+0000 7f22edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:35.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:35.694+0000 7f22edffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:36.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.020+0000 7f22ff82a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:36.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.020+0000 7f22ff82a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:36.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.020+0000 7f22ff82a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:36.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.020+0000 7f22ff82a640 1 -- 172.21.15.67:0/2380556575 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22d8005330 con 0x7f22f80b07d0 2024-03-29T11:27:36.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.022+0000 7f22edffb640 1 -- 172.21.15.67:0/2380556575 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f22f40509e0 con 0x7f22f80b07d0 2024-03-29T11:27:36.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.022+0000 7f22edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:36.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.022+0000 7f22edffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:36.010 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:27:36.011 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:36.015 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.026+0000 7f22e37fe640 10 librados: watch_flush enter 2024-03-29T11:27:36.015 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.026+0000 7f22e37fe640 10 librados: watch_flush exit 2024-03-29T11:27:36.015 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.026+0000 7f22e37fe640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-29T11:27:36.015 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.026+0000 7f22e37fe640 10 client.4562.objecter successfully canceled tick 2024-03-29T11:27:36.015 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.027+0000 7f22e37fe640 1 -- 172.21.15.67:0/2380556575 >> v1:172.21.15.67:6816/404787197 conn(0x7f22dc06e1e0 legacy=0x7f22dc0706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.015 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.027+0000 7f22e37fe640 10 monclient: shutdown 2024-03-29T11:27:36.015 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.027+0000 7f22e37fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:36.015 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.027+0000 7f22e37fe640 1 -- 172.21.15.67:0/2380556575 >> v1:172.21.15.67:6790/0 conn(0x7f22f80b07d0 legacy=0x7f22f81e4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.015 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.027+0000 7f22e37fe640 1 -- 172.21.15.67:0/2380556575 shutdown_connections 2024-03-29T11:27:36.015 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.027+0000 7f22e37fe640 1 -- 172.21.15.67:0/2380556575 >> 172.21.15.67:0/2380556575 conn(0x7f22f806b660 msgr2=0x7f22f813ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:36.019 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.028+0000 7f22e37fe640 1 -- 172.21.15.67:0/2380556575 shutdown_connections 2024-03-29T11:27:36.019 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.029+0000 7f22e37fe640 1 -- 172.21.15.67:0/2380556575 wait complete. 2024-03-29T11:27:36.019 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:36.031+0000 7f22e37fe640 1 librados: shutdown 2024-03-29T11:27:36.037 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:42.845 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:42.983 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.995+0000 7f9e5e233640 10 monclient: get_monmap_and_config 2024-03-29T11:27:42.983 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.995+0000 7f9e5e233640 10 monclient: build_initial_monmap 2024-03-29T11:27:42.983 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.995+0000 7f9e5e233640 10 monclient: monmap: 2024-03-29T11:27:42.984 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:42.984 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:42.984 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:27:42.996014+0000 2024-03-29T11:27:42.984 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:27:42.996014+0000 2024-03-29T11:27:42.984 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:42.984 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:42.984 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:42.984 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:27:42.984 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:27:42.984 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:42.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.996+0000 7f9e5e233640 1 Processor -- start 2024-03-29T11:27:42.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.996+0000 7f9e5e233640 1 -- start start 2024-03-29T11:27:42.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.996+0000 7f9e5e233640 10 monclient: init 2024-03-29T11:27:42.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.997+0000 7f9e5e233640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:42.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.997+0000 7f9e5e233640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:42.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.997+0000 7f9e5e233640 10 monclient(hunting): picked mon.noname-c con 0x7f9e50006ec0 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:42.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.997+0000 7f9e5e233640 10 monclient(hunting): picked mon.noname-b con 0x7f9e50033530 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:42.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.997+0000 7f9e5e233640 10 monclient(hunting): picked mon.noname-a con 0x7f9e500cfcb0 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:42.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.997+0000 7f9e5e233640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e500cf110 con 0x7f9e500cfcb0 2024-03-29T11:27:42.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.997+0000 7f9e5e233640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5002c1d0 con 0x7f9e50006ec0 2024-03-29T11:27:42.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.997+0000 7f9e5e233640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e500cbb90 con 0x7f9e50033530 2024-03-29T11:27:42.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.997+0000 7f9e5e233640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:42.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.997+0000 7f9e5e233640 10 monclient(hunting): authenticate will time out at 1323.293151s 2024-03-29T11:27:42.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.997+0000 7f9e5ca30640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f9e50033530 0x7f9e500cf5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:34394/0 (socket says 172.21.15.67:34394) 2024-03-29T11:27:42.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.997+0000 7f9e5ca30640 1 -- 172.21.15.67:0/4143824784 learned_addr learned my addr 172.21.15.67:0/4143824784 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:42.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.998+0000 7f9e4ffff640 1 -- 172.21.15.67:0/4143824784 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021914368 0 0) 0x7f9e500cbb90 con 0x7f9e50033530 2024-03-29T11:27:42.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.998+0000 7f9e4ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:42.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.998+0000 7f9e4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:42.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.998+0000 7f9e4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:42.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.998+0000 7f9e4ffff640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:27:42.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.998+0000 7f9e4ffff640 1 -- 172.21.15.67:0/4143824784 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e44003520 con 0x7f9e50033530 2024-03-29T11:27:42.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.999+0000 7f9e4ffff640 1 -- 172.21.15.67:0/4143824784 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 72557050 0 0) 0x7f9e44003520 con 0x7f9e50033530 2024-03-29T11:27:42.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:42.999+0000 7f9e4ffff640 1 -- 172.21.15.67:0/4143824784 >> v1:172.21.15.82:6789/0 conn(0x7f9e50006ec0 legacy=0x7f9e50032e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:42.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.002+0000 7f9e4ffff640 1 -- 172.21.15.67:0/4143824784 >> v1:172.21.15.67:6789/0 conn(0x7f9e500cfcb0 legacy=0x7f9e500d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.002+0000 7f9e4ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:42.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.002+0000 7f9e4ffff640 1 monclient: found mon.noname-b 2024-03-29T11:27:42.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.002+0000 7f9e4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:42.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.002+0000 7f9e4ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:27:42.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.002+0000 7f9e4ffff640 1 -- 172.21.15.67:0/4143824784 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e50029c30 con 0x7f9e50033530 2024-03-29T11:27:42.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.002+0000 7f9e4ffff640 10 monclient: _finish_auth 0 2024-03-29T11:27:42.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.002+0000 7f9e4ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:42.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.002+0000 7f9e4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:42.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.002+0000 7f9e4ffff640 1 -- 172.21.15.67:0/4143824784 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f9e48001690 con 0x7f9e50033530 2024-03-29T11:27:42.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.002+0000 7f9e4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:42.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.003+0000 7f9e4ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:42.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.003+0000 7f9e4ffff640 10 monclient: dump: 2024-03-29T11:27:42.994 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:42.994 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:42.994 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:42.994 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:42.994 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:42.994 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:42.994 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:42.994 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:42.994 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:42.994 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:42.994 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.003+0000 7f9e4ffff640 1 -- 172.21.15.67:0/4143824784 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e48001860 con 0x7f9e50033530 2024-03-29T11:27:42.994 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.003+0000 7f9e4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:42.994 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.003+0000 7f9e4ffff640 1 -- 172.21.15.67:0/4143824784 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f9e48001c70 con 0x7f9e50033530 2024-03-29T11:27:42.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.003+0000 7f9e5e233640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:27:42.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.003+0000 7f9e5e233640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:42.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.003+0000 7f9e5e233640 4 set_mon_vals no callback set 2024-03-29T11:27:42.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.003+0000 7f9e4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:42.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.003+0000 7f9e4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:42.995 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.003+0000 7f9e4ffff640 10 monclient: dump: 2024-03-29T11:27:42.995 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:42.996 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:42.996 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:42.997 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:42.997 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:42.997 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:42.997 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:42.997 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:42.997 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:42.997 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:42.997 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.004+0000 7f9e5e233640 10 monclient: shutdown 2024-03-29T11:27:42.997 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.004+0000 7f9e5e233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:42.997 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.004+0000 7f9e5e233640 1 -- 172.21.15.67:0/4143824784 >> v1:172.21.15.67:6790/0 conn(0x7f9e50033530 legacy=0x7f9e500cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.005+0000 7f9e5e233640 1 -- 172.21.15.67:0/4143824784 shutdown_connections 2024-03-29T11:27:43.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.006+0000 7f9e5e233640 1 -- 172.21.15.67:0/4143824784 >> 172.21.15.67:0/4143824784 conn(0x7f9e50005d40 msgr2=0x7f9e50006150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:43.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.006+0000 7f9e5e233640 1 -- 172.21.15.67:0/4143824784 shutdown_connections 2024-03-29T11:27:43.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.006+0000 7f9e5e233640 1 -- 172.21.15.67:0/4143824784 wait complete. 2024-03-29T11:27:43.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.007+0000 7f9e5e233640 10 monclient: build_initial_monmap 2024-03-29T11:27:43.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.007+0000 7f9e5e233640 10 monclient: monmap: 2024-03-29T11:27:43.002 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:43.002 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:43.002 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:27:43.003 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:27:43.003 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:43.003 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:43.003 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:43.003 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:27:43.003 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:27:43.003 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:43.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.007+0000 7f9e5e233640 1 librados: starting msgr at 2024-03-29T11:27:43.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.007+0000 7f9e5e233640 1 librados: starting objecter 2024-03-29T11:27:43.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.007+0000 7f9e5e233640 1 Processor -- start 2024-03-29T11:27:43.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.007+0000 7f9e5e233640 1 -- start start 2024-03-29T11:27:43.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.007+0000 7f9e5e233640 1 librados: setting wanted keys 2024-03-29T11:27:43.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.007+0000 7f9e5e233640 1 librados: calling monclient init 2024-03-29T11:27:43.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.007+0000 7f9e5e233640 10 monclient: init 2024-03-29T11:27:43.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.008+0000 7f9e5e233640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:43.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.008+0000 7f9e5e233640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:43.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.008+0000 7f9e5e233640 10 monclient(hunting): picked mon.noname-a con 0x7f9e50006ec0 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:43.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.008+0000 7f9e5e233640 10 monclient(hunting): picked mon.noname-c con 0x7f9e500cfcb0 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:43.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.008+0000 7f9e5e233640 10 monclient(hunting): picked mon.noname-b con 0x7f9e5002e7e0 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:43.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.008+0000 7f9e5e233640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e50164490 con 0x7f9e50006ec0 2024-03-29T11:27:43.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.008+0000 7f9e5e233640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e50032910 con 0x7f9e5002e7e0 2024-03-29T11:27:43.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.011+0000 7f9e5ca30640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f9e500cfcb0 0x7f9e50164920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:35380/0 (socket says 172.21.15.67:35380) 2024-03-29T11:27:43.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.011+0000 7f9e5ca30640 1 -- 172.21.15.67:0/2172529076 learned_addr learned my addr 172.21.15.67:0/2172529076 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:43.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.011+0000 7f9e5e233640 1 -- 172.21.15.67:0/2172529076 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e50165030 con 0x7f9e500cfcb0 2024-03-29T11:27:43.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.011+0000 7f9e5e233640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:43.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.011+0000 7f9e5e233640 10 monclient(hunting): authenticate will time out at 1323.307444s 2024-03-29T11:27:43.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.011+0000 7f9e4dffb640 10 client.?.objecter ms_handle_connect 0x7f9e500cfcb0 2024-03-29T11:27:43.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.011+0000 7f9e4dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:43.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.011+0000 7f9e4dffb640 10 client.?.objecter ms_handle_connect 0x7f9e5002e7e0 2024-03-29T11:27:43.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.011+0000 7f9e4dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:43.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.011+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920873478 0 0) 0x7f9e50165030 con 0x7f9e500cfcb0 2024-03-29T11:27:43.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:43.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:43.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:43.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:27:43.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e3c004eb0 con 0x7f9e500cfcb0 2024-03-29T11:27:43.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648327133 0 0) 0x7f9e50032910 con 0x7f9e5002e7e0 2024-03-29T11:27:43.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:43.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:43.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:43.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:27:43.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e50165030 con 0x7f9e5002e7e0 2024-03-29T11:27:43.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3817912020 0 0) 0x7f9e3c004eb0 con 0x7f9e500cfcb0 2024-03-29T11:27:43.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e3c005490 con 0x7f9e500cfcb0 2024-03-29T11:27:43.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f9e48001b70 con 0x7f9e500cfcb0 2024-03-29T11:27:43.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:43.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 10 monclient(hunting): dump: 2024-03-29T11:27:43.006 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:43.006 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:43.006 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:43.006 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:43.006 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2909999674 0 0) 0x7f9e50165030 con 0x7f9e5002e7e0 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e3c004eb0 con 0x7f9e5002e7e0 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f9e540016c0 con 0x7f9e5002e7e0 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.012+0000 7f9e4dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 monclient(hunting): dump: 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:43.007 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:43.008 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:43.008 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.008 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:43.008 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:43.008 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:43.008 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:43.008 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:43.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 client.?.objecter ms_handle_connect 0x7f9e50006ec0 2024-03-29T11:27:43.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:43.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571748756 0 0) 0x7f9e50164490 con 0x7f9e50006ec0 2024-03-29T11:27:43.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:43.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:43.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:43.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:27:43.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e3c005d50 con 0x7f9e50006ec0 2024-03-29T11:27:43.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1635621143 0 0) 0x7f9e3c005490 con 0x7f9e500cfcb0 2024-03-29T11:27:43.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 >> v1:172.21.15.82:6789/0 conn(0x7f9e5002e7e0 legacy=0x7f9e500c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 >> v1:172.21.15.67:6789/0 conn(0x7f9e50006ec0 legacy=0x7f9e500312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:43.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 1 monclient: found mon.c 2024-03-29T11:27:43.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:43.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:43.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e501651c0 con 0x7f9e500cfcb0 2024-03-29T11:27:43.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:43.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:43.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:43.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e5e233640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:27:43.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e5e233640 10 monclient: _renew_subs 2024-03-29T11:27:43.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e5e233640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:43.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e48004bf0 con 0x7f9e500cfcb0 2024-03-29T11:27:43.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e5e233640 1 -- 172.21.15.67:0/2172529076 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e500c73e0 con 0x7f9e500cfcb0 2024-03-29T11:27:43.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:43.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f9e48004d60 con 0x7f9e500cfcb0 2024-03-29T11:27:43.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4f7fe640 4 set_mon_vals no callback set 2024-03-29T11:27:43.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:43.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.013+0000 7f9e4dffb640 10 monclient: dump: 2024-03-29T11:27:43.010 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:43.010 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:43.010 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:43.010 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:43.011 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.011 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:43.011 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:43.011 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:43.011 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:43.011 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:43.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.014+0000 7f9e5e233640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:43.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.014+0000 7f9e5e233640 10 monclient: _renew_subs 2024-03-29T11:27:43.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.014+0000 7f9e5e233640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:43.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.014+0000 7f9e5e233640 1 -- 172.21.15.67:0/2172529076 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e500321a0 con 0x7f9e500cfcb0 2024-03-29T11:27:43.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.014+0000 7f9e5e233640 1 librados: init done 2024-03-29T11:27:43.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.015+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f9e4801ad30 con 0x7f9e500cfcb0 2024-03-29T11:27:43.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.016+0000 7f9e5e233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:43.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.016+0000 7f9e5e233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:43.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.016+0000 7f9e5e233640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:43.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.016+0000 7f9e5e233640 1 -- 172.21.15.67:0/2172529076 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e500072a0 con 0x7f9e500cfcb0 2024-03-29T11:27:43.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.016+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f9e48084980 con 0x7f9e500cfcb0 2024-03-29T11:27:43.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.016+0000 7f9e4dffb640 10 client.4568.objecter ms_dispatch 0x7f9e58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:43.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.016+0000 7f9e4dffb640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:43.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.016+0000 7f9e4dffb640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:43.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.016+0000 7f9e4dffb640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T11:27:43.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.017+0000 7f9e5d231640 10 monclient: get_auth_request con 0x7f9e3c06e2e0 auth_method 0 2024-03-29T11:27:43.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.017+0000 7f9e4dffb640 10 client.4568.objecter ms_handle_connect 0x7f9e3c06e2e0 2024-03-29T11:27:43.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.024+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9e48050260 con 0x7f9e500cfcb0 2024-03-29T11:27:43.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.024+0000 7f9e4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:43.014 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.024+0000 7f9e4dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:43.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.335+0000 7f9e5e233640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.335+0000 7f9e5e233640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.335+0000 7f9e5e233640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:43.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.335+0000 7f9e5e233640 1 -- 172.21.15.67:0/2172529076 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e50005120 con 0x7f9e500cfcb0 2024-03-29T11:27:43.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.335+0000 7f9e4dffb640 1 -- 172.21.15.67:0/2172529076 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f9e48052190 con 0x7f9e500cfcb0 2024-03-29T11:27:43.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.335+0000 7f9e4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.335+0000 7f9e4dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:43.324 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:27:43.325 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:43.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.339+0000 7f9e5e233640 10 librados: watch_flush enter 2024-03-29T11:27:43.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.339+0000 7f9e5e233640 10 librados: watch_flush exit 2024-03-29T11:27:43.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.339+0000 7f9e5e233640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T11:27:43.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.339+0000 7f9e5e233640 10 client.4568.objecter successfully canceled tick 2024-03-29T11:27:43.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.339+0000 7f9e5e233640 1 -- 172.21.15.67:0/2172529076 >> v1:172.21.15.67:6816/404787197 conn(0x7f9e3c06e2e0 legacy=0x7f9e3c0707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.339+0000 7f9e5e233640 10 monclient: shutdown 2024-03-29T11:27:43.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.339+0000 7f9e5e233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:43.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.339+0000 7f9e5e233640 1 -- 172.21.15.67:0/2172529076 >> v1:172.21.15.67:6790/0 conn(0x7f9e500cfcb0 legacy=0x7f9e50164920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.328 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.339+0000 7f9e5e233640 1 -- 172.21.15.67:0/2172529076 shutdown_connections 2024-03-29T11:27:43.328 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.339+0000 7f9e5e233640 1 -- 172.21.15.67:0/2172529076 >> 172.21.15.67:0/2172529076 conn(0x7f9e50005d40 msgr2=0x7f9e50037ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:43.328 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.339+0000 7f9e5e233640 1 -- 172.21.15.67:0/2172529076 shutdown_connections 2024-03-29T11:27:43.328 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.339+0000 7f9e5e233640 1 -- 172.21.15.67:0/2172529076 wait complete. 2024-03-29T11:27:43.328 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:43.340+0000 7f9e5e233640 1 librados: shutdown 2024-03-29T11:27:43.345 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:50.154 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:50.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.333+0000 7f36fbfff640 10 monclient: get_monmap_and_config 2024-03-29T11:27:50.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.333+0000 7f36fbfff640 10 monclient: build_initial_monmap 2024-03-29T11:27:50.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.333+0000 7f36fbfff640 10 monclient: monmap: 2024-03-29T11:27:50.322 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:50.322 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:50.322 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:27:50.334177+0000 2024-03-29T11:27:50.322 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:27:50.334177+0000 2024-03-29T11:27:50.322 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:50.322 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:50.323 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:50.323 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:27:50.323 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:27:50.323 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:50.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.335+0000 7f36fbfff640 1 Processor -- start 2024-03-29T11:27:50.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.335+0000 7f36fbfff640 1 -- start start 2024-03-29T11:27:50.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.335+0000 7f36fbfff640 10 monclient: init 2024-03-29T11:27:50.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.336+0000 7f36fbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:50.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.336+0000 7f36fbfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:50.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.336+0000 7f36fbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f36fc0b4920 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:50.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.336+0000 7f36fbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f36fc0b4200 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:50.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.336+0000 7f36fbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f36fc13f9a0 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:50.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.336+0000 7f36fbfff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36fc0b0930 con 0x7f36fc0b4200 2024-03-29T11:27:50.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.336+0000 7f36fbfff640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36fc0af460 con 0x7f36fc0b4920 2024-03-29T11:27:50.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.336+0000 7f36fbfff640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36fc0aeda0 con 0x7f36fc13f9a0 2024-03-29T11:27:50.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.336+0000 7f36fbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:50.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.336+0000 7f36fbfff640 10 monclient(hunting): authenticate will time out at 1330.632538s 2024-03-29T11:27:50.327 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.339+0000 7f36fb7fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f36fc13f9a0 0x7f36fc14d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:36672/0 (socket says 172.21.15.67:36672) 2024-03-29T11:27:50.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.339+0000 7f36fb7fe640 1 -- 172.21.15.67:0/4031629271 learned_addr learned my addr 172.21.15.67:0/4031629271 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:50.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.340+0000 7f36f9ffb640 1 -- 172.21.15.67:0/4031629271 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956172130 0 0) 0x7f36fc0af460 con 0x7f36fc0b4920 2024-03-29T11:27:50.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.340+0000 7f36f9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.340+0000 7f36f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.340+0000 7f36f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.340+0000 7f36f9ffb640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:27:50.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.340+0000 7f36f9ffb640 1 -- 172.21.15.67:0/4031629271 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e8003520 con 0x7f36fc0b4920 2024-03-29T11:27:50.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 1 -- 172.21.15.67:0/4031629271 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2063019138 0 0) 0x7f36e8003520 con 0x7f36fc0b4920 2024-03-29T11:27:50.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 1 -- 172.21.15.67:0/4031629271 >> v1:172.21.15.67:6790/0 conn(0x7f36fc13f9a0 legacy=0x7f36fc14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 1 -- 172.21.15.67:0/4031629271 >> v1:172.21.15.67:6789/0 conn(0x7f36fc0b4200 legacy=0x7f36fc0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:50.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:50.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:27:50.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:50.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:27:50.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 1 -- 172.21.15.67:0/4031629271 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36fc0af980 con 0x7f36fc0b4920 2024-03-29T11:27:50.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:50.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:50.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:50.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 1 -- 172.21.15.67:0/4031629271 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f36e40015a0 con 0x7f36fc0b4920 2024-03-29T11:27:50.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:50.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 10 monclient: dump: 2024-03-29T11:27:50.331 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:50.331 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:50.331 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:50.332 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:50.332 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.332 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:50.332 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:50.332 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:50.332 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:50.332 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36fbfff640 5 monclient: authenticate success, global_id 4471 2024-03-29T11:27:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36fbfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 1 -- 172.21.15.67:0/4031629271 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36e4001830 con 0x7f36fc0b4920 2024-03-29T11:27:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36fbfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:50.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.341+0000 7f36fbfff640 4 set_mon_vals no callback set 2024-03-29T11:27:50.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.342+0000 7f36fbfff640 10 monclient: shutdown 2024-03-29T11:27:50.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.342+0000 7f36fbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:50.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.342+0000 7f36fbfff640 1 -- 172.21.15.67:0/4031629271 >> v1:172.21.15.82:6789/0 conn(0x7f36fc0b4920 legacy=0x7f36fc13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.342+0000 7f36f9ffb640 1 -- 172.21.15.67:0/4031629271 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f36e40015a0 con 0x7f36fc0b4920 2024-03-29T11:27:50.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.342+0000 7f36f9ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:27:50.333 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.343+0000 7f36fbfff640 1 -- 172.21.15.67:0/4031629271 shutdown_connections 2024-03-29T11:27:50.334 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.343+0000 7f36fbfff640 1 -- 172.21.15.67:0/4031629271 >> 172.21.15.67:0/4031629271 conn(0x7f36fc06d230 msgr2=0x7f36fc066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:50.334 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.344+0000 7f36fbfff640 1 -- 172.21.15.67:0/4031629271 shutdown_connections 2024-03-29T11:27:50.335 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.346+0000 7f36fbfff640 1 -- 172.21.15.67:0/4031629271 wait complete. 2024-03-29T11:27:50.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.347+0000 7f36fbfff640 10 monclient: build_initial_monmap 2024-03-29T11:27:50.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.347+0000 7f36fbfff640 10 monclient: monmap: 2024-03-29T11:27:50.337 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:50.337 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:50.337 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:27:50.337 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:27:50.337 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:50.337 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:50.337 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:50.337 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:27:50.337 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:27:50.337 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:50.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.348+0000 7f36fbfff640 1 librados: starting msgr at 2024-03-29T11:27:50.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.348+0000 7f36fbfff640 1 librados: starting objecter 2024-03-29T11:27:50.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.348+0000 7f36fbfff640 1 Processor -- start 2024-03-29T11:27:50.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.348+0000 7f36fbfff640 1 -- start start 2024-03-29T11:27:50.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.348+0000 7f36fbfff640 1 librados: setting wanted keys 2024-03-29T11:27:50.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.348+0000 7f36fbfff640 1 librados: calling monclient init 2024-03-29T11:27:50.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.348+0000 7f36fbfff640 10 monclient: init 2024-03-29T11:27:50.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.349+0000 7f36fbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:50.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.349+0000 7f36fbfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:50.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.349+0000 7f36fbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f36fc0b4200 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:50.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.349+0000 7f36fbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f36fc0b4920 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:50.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.349+0000 7f36fbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f36fc13f9a0 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:50.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.349+0000 7f36fbfff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36fc1451b0 con 0x7f36fc0b4920 2024-03-29T11:27:50.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.349+0000 7f36fbfff640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36fc137f20 con 0x7f36fc0b4200 2024-03-29T11:27:50.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.349+0000 7f36fbfff640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36fc145640 con 0x7f36fc13f9a0 2024-03-29T11:27:50.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.349+0000 7f36fbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:50.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.349+0000 7f36fbfff640 10 monclient(hunting): authenticate will time out at 1330.645631s 2024-03-29T11:27:50.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.353+0000 7f36faffd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f36fc0b4200 0x7f36fc1db4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:57679/0 (socket says 172.21.15.67:57679) 2024-03-29T11:27:50.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.353+0000 7f36faffd640 1 -- 172.21.15.67:0/2041831965 learned_addr learned my addr 172.21.15.67:0/2041831965 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:50.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 10 client.?.objecter ms_handle_connect 0x7f36fc0b4200 2024-03-29T11:27:50.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:50.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 10 client.?.objecter ms_handle_connect 0x7f36fc13f9a0 2024-03-29T11:27:50.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:50.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070122100 0 0) 0x7f36fc145640 con 0x7f36fc13f9a0 2024-03-29T11:27:50.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:27:50.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36dc004eb0 con 0x7f36fc13f9a0 2024-03-29T11:27:50.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985044942 0 0) 0x7f36fc137f20 con 0x7f36fc0b4200 2024-03-29T11:27:50.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.355+0000 7f36f3fff640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:27:50.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36fc145640 con 0x7f36fc0b4200 2024-03-29T11:27:50.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 10 client.?.objecter ms_handle_connect 0x7f36fc0b4920 2024-03-29T11:27:50.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:50.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900008919 0 0) 0x7f36fc1451b0 con 0x7f36fc0b4920 2024-03-29T11:27:50.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:27:50.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36fc137f20 con 0x7f36fc0b4920 2024-03-29T11:27:50.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2012886644 0 0) 0x7f36fc145640 con 0x7f36fc0b4200 2024-03-29T11:27:50.349 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36fc1451b0 con 0x7f36fc0b4200 2024-03-29T11:27:50.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f36e40015a0 con 0x7f36fc0b4200 2024-03-29T11:27:50.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:50.350 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 10 monclient(hunting): dump: 2024-03-29T11:27:50.350 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:50.350 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:50.350 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:50.350 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:50.350 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.350 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:50.350 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:50.350 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:50.350 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:50.350 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:50.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:50.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1547254864 0 0) 0x7f36dc004eb0 con 0x7f36fc13f9a0 2024-03-29T11:27:50.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36fc145640 con 0x7f36fc13f9a0 2024-03-29T11:27:50.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f36f4001700 con 0x7f36fc13f9a0 2024-03-29T11:27:50.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:50.351 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.356+0000 7f36f3fff640 10 monclient(hunting): dump: 2024-03-29T11:27:50.351 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:50.351 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:50.351 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:50.351 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:50.351 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.351 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:50.351 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:50.352 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:50.352 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:50.352 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:50.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.358+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1001720465 0 0) 0x7f36fc1451b0 con 0x7f36fc0b4200 2024-03-29T11:27:50.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.358+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 >> v1:172.21.15.67:6790/0 conn(0x7f36fc13f9a0 legacy=0x7f36fc137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.359+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 >> v1:172.21.15.67:6789/0 conn(0x7f36fc0b4920 legacy=0x7f36fc137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.359+0000 7f36f3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:50.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.359+0000 7f36f3fff640 1 monclient: found mon.b 2024-03-29T11:27:50.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.359+0000 7f36f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:50.352 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.359+0000 7f36f3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:27:50.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.360+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36fc06e1d0 con 0x7f36fc0b4200 2024-03-29T11:27:50.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.360+0000 7f36f3fff640 10 monclient: _finish_auth 0 2024-03-29T11:27:50.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.360+0000 7f36f3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:50.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.360+0000 7f36f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:50.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.360+0000 7f36fbfff640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:27:50.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.360+0000 7f36fbfff640 10 monclient: _renew_subs 2024-03-29T11:27:50.355 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.360+0000 7f36fbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:27:50.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.360+0000 7f36fbfff640 1 -- 172.21.15.67:0/2041831965 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36fc1462a0 con 0x7f36fc0b4200 2024-03-29T11:27:50.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.360+0000 7f36fbfff640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:50.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.360+0000 7f36fbfff640 10 monclient: _renew_subs 2024-03-29T11:27:50.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.360+0000 7f36fbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:27:50.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.360+0000 7f36fbfff640 1 -- 172.21.15.67:0/2041831965 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36fc1445e0 con 0x7f36fc0b4200 2024-03-29T11:27:50.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.360+0000 7f36fbfff640 1 librados: init done 2024-03-29T11:27:50.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.361+0000 7f36fbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:50.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.362+0000 7f36fbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:50.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.362+0000 7f36fbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:27:50.356 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.362+0000 7f36fbfff640 1 -- 172.21.15.67:0/2041831965 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36fc04f730 con 0x7f36fc0b4200 2024-03-29T11:27:50.357 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.362+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36e40038a0 con 0x7f36fc0b4200 2024-03-29T11:27:50.357 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.362+0000 7f36f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:50.357 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.362+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f36e4003c30 con 0x7f36fc0b4200 2024-03-29T11:27:50.357 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.362+0000 7f36f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.357 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.362+0000 7f36f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:50.357 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.362+0000 7f36f3fff640 10 monclient: dump: 2024-03-29T11:27:50.357 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:50.357 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:50.357 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:50.358 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:50.358 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.358 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:50.358 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:50.358 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:50.358 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:50.358 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:50.358 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.363+0000 7f36f8ff9640 4 set_mon_vals no callback set 2024-03-29T11:27:50.358 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.365+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f36e4003d90 con 0x7f36fc0b4200 2024-03-29T11:27:50.358 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.366+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f36e4084b10 con 0x7f36fc0b4200 2024-03-29T11:27:50.358 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.366+0000 7f36f3fff640 10 client.4474.objecter ms_dispatch 0x7f36fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:50.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.366+0000 7f36f3fff640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:50.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.366+0000 7f36f3fff640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:50.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.366+0000 7f36f3fff640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T11:27:50.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.366+0000 7f36fa7fc640 10 monclient: get_auth_request con 0x7f36dc06de40 auth_method 0 2024-03-29T11:27:50.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.367+0000 7f36f3fff640 10 client.4474.objecter ms_handle_connect 0x7f36dc06de40 2024-03-29T11:27:50.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.369+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36e40503f0 con 0x7f36fc0b4200 2024-03-29T11:27:50.359 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.369+0000 7f36f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:50.360 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.369+0000 7f36f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:50.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.723+0000 7f36fbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:50.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.723+0000 7f36fbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:50.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.723+0000 7f36fbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:27:50.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.723+0000 7f36fbfff640 1 -- 172.21.15.67:0/2041831965 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36fc13a1a0 con 0x7f36fc0b4200 2024-03-29T11:27:50.713 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.724+0000 7f36f3fff640 1 -- 172.21.15.67:0/2041831965 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f36e4052320 con 0x7f36fc0b4200 2024-03-29T11:27:50.713 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.724+0000 7f36f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:50.713 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.724+0000 7f36f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:50.713 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:27:50.713 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:50.717 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.728+0000 7f36fbfff640 10 librados: watch_flush enter 2024-03-29T11:27:50.717 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.728+0000 7f36fbfff640 10 librados: watch_flush exit 2024-03-29T11:27:50.717 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.728+0000 7f36fbfff640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T11:27:50.717 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.728+0000 7f36fbfff640 10 client.4474.objecter successfully canceled tick 2024-03-29T11:27:50.718 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.728+0000 7f36fbfff640 1 -- 172.21.15.67:0/2041831965 >> v1:172.21.15.67:6816/404787197 conn(0x7f36dc06de40 legacy=0x7f36dc070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.718 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.729+0000 7f36fbfff640 10 monclient: shutdown 2024-03-29T11:27:50.718 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.729+0000 7f36fbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:50.718 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.729+0000 7f36fbfff640 1 -- 172.21.15.67:0/2041831965 >> v1:172.21.15.82:6789/0 conn(0x7f36fc0b4200 legacy=0x7f36fc1db4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.718 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.729+0000 7f36fbfff640 1 -- 172.21.15.67:0/2041831965 shutdown_connections 2024-03-29T11:27:50.718 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.729+0000 7f36fbfff640 1 -- 172.21.15.67:0/2041831965 >> 172.21.15.67:0/2041831965 conn(0x7f36fc06d230 msgr2=0x7f36fc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:50.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.729+0000 7f36fbfff640 1 -- 172.21.15.67:0/2041831965 shutdown_connections 2024-03-29T11:27:50.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.729+0000 7f36fbfff640 1 -- 172.21.15.67:0/2041831965 wait complete. 2024-03-29T11:27:50.719 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:50.730+0000 7f36fbfff640 1 librados: shutdown 2024-03-29T11:27:50.735 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:57.543 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:57.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.695+0000 7fea647c8640 10 monclient: get_monmap_and_config 2024-03-29T11:27:57.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.695+0000 7fea647c8640 10 monclient: build_initial_monmap 2024-03-29T11:27:57.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.696+0000 7fea647c8640 10 monclient: monmap: 2024-03-29T11:27:57.686 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:57.686 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:57.686 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:27:57.697796+0000 2024-03-29T11:27:57.686 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:27:57.697796+0000 2024-03-29T11:27:57.686 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:57.686 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:57.686 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:57.687 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:27:57.687 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:27:57.687 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:57.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.697+0000 7fea647c8640 1 Processor -- start 2024-03-29T11:27:57.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.697+0000 7fea647c8640 1 -- start start 2024-03-29T11:27:57.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.697+0000 7fea647c8640 10 monclient: init 2024-03-29T11:27:57.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.698+0000 7fea647c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:57.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.698+0000 7fea647c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:57.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.698+0000 7fea647c8640 10 monclient(hunting): picked mon.noname-a con 0x7fea5c06ed30 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:57.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.698+0000 7fea647c8640 10 monclient(hunting): picked mon.noname-c con 0x7fea5c149ae0 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:57.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.698+0000 7fea647c8640 10 monclient(hunting): picked mon.noname-b con 0x7fea5c14a5b0 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:57.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.698+0000 7fea647c8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5c14aa20 con 0x7fea5c06ed30 2024-03-29T11:27:57.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.698+0000 7fea647c8640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5c070260 con 0x7fea5c149ae0 2024-03-29T11:27:57.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.698+0000 7fea62564640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fea5c06ed30 0x7fea5c0b0340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55204/0 (socket says 172.21.15.67:55204) 2024-03-29T11:27:57.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.698+0000 7fea62564640 1 -- 172.21.15.67:0/439402458 learned_addr learned my addr 172.21.15.67:0/439402458 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:57.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.698+0000 7fea62d65640 1 --1- 172.21.15.67:0/439402458 >> v1:172.21.15.67:6790/0 conn(0x7fea5c14a5b0 0x7fea5c151e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:49070/0 (socket says 172.21.15.67:49070) 2024-03-29T11:27:57.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea647c8640 1 -- 172.21.15.67:0/439402458 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5c070540 con 0x7fea5c14a5b0 2024-03-29T11:27:57.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea647c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:57.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea647c8640 10 monclient(hunting): authenticate will time out at 1337.995985s 2024-03-29T11:27:57.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 1 -- 172.21.15.67:0/439402458 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3674833257 0 0) 0x7fea5c070540 con 0x7fea5c14a5b0 2024-03-29T11:27:57.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient(hunting): _negotiate 2024-03-29T11:27:57.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:57.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:57.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:27:57.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 1 -- 172.21.15.67:0/439402458 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea48003520 con 0x7fea5c14a5b0 2024-03-29T11:27:57.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 1 -- 172.21.15.67:0/439402458 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662502747 0 0) 0x7fea5c14aa20 con 0x7fea5c06ed30 2024-03-29T11:27:57.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient(hunting): _negotiate 2024-03-29T11:27:57.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:57.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:57.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:27:57.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 1 -- 172.21.15.67:0/439402458 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea5c070540 con 0x7fea5c06ed30 2024-03-29T11:27:57.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 1 -- 172.21.15.67:0/439402458 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1393844599 0 0) 0x7fea48003520 con 0x7fea5c14a5b0 2024-03-29T11:27:57.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 1 -- 172.21.15.67:0/439402458 >> v1:172.21.15.82:6789/0 conn(0x7fea5c149ae0 legacy=0x7fea5c071ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:57.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 1 -- 172.21.15.67:0/439402458 >> v1:172.21.15.67:6789/0 conn(0x7fea5c06ed30 legacy=0x7fea5c0b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:57.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient: _finish_hunting 0 2024-03-29T11:27:57.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 1 monclient: found mon.noname-b 2024-03-29T11:27:57.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:57.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:27:57.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 1 -- 172.21.15.67:0/439402458 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c14a090 con 0x7fea5c14a5b0 2024-03-29T11:27:57.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient: _finish_auth 0 2024-03-29T11:27:57.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient: _check_auth_tickets 2024-03-29T11:27:57.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:57.695 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 1 -- 172.21.15.67:0/439402458 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fea54001690 con 0x7fea5c14a5b0 2024-03-29T11:27:57.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:57.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:57.696 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.699+0000 7fea61562640 10 monclient: dump: 2024-03-29T11:27:57.696 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:57.696 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:57.696 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:57.696 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:57.696 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:57.696 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:57.696 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:57.696 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:57.696 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:57.696 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:57.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.700+0000 7fea647c8640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:27:57.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.700+0000 7fea647c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:57.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.700+0000 7fea61562640 1 -- 172.21.15.67:0/439402458 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea540018b0 con 0x7fea5c14a5b0 2024-03-29T11:27:57.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.700+0000 7fea61562640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:57.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.700+0000 7fea647c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:57.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.700+0000 7fea647c8640 4 set_mon_vals no callback set 2024-03-29T11:27:57.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.700+0000 7fea61562640 1 -- 172.21.15.67:0/439402458 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fea54001c40 con 0x7fea5c14a5b0 2024-03-29T11:27:57.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.700+0000 7fea61562640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:57.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.700+0000 7fea61562640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:57.697 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.700+0000 7fea61562640 10 monclient: dump: 2024-03-29T11:27:57.697 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:57.697 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:57.697 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:57.698 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:57.698 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:57.698 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:57.698 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:57.698 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:57.698 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:57.698 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:57.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.701+0000 7fea647c8640 10 monclient: shutdown 2024-03-29T11:27:57.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.701+0000 7fea647c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:57.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.701+0000 7fea647c8640 1 -- 172.21.15.67:0/439402458 >> v1:172.21.15.67:6790/0 conn(0x7fea5c14a5b0 legacy=0x7fea5c151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:57.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.701+0000 7fea647c8640 1 -- 172.21.15.67:0/439402458 shutdown_connections 2024-03-29T11:27:57.698 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.701+0000 7fea647c8640 1 -- 172.21.15.67:0/439402458 >> 172.21.15.67:0/439402458 conn(0x7fea5c0b1430 msgr2=0x7fea5c0b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:57.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.701+0000 7fea647c8640 1 -- 172.21.15.67:0/439402458 shutdown_connections 2024-03-29T11:27:57.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.701+0000 7fea647c8640 1 -- 172.21.15.67:0/439402458 wait complete. 2024-03-29T11:27:57.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.702+0000 7fea647c8640 10 monclient: build_initial_monmap 2024-03-29T11:27:57.699 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.702+0000 7fea647c8640 10 monclient: monmap: 2024-03-29T11:27:57.699 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:27:57.699 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:57.699 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:27:57.699 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:27:57.699 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:57.699 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:57.699 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:27:57.700 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:27:57.700 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:27:57.700 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:57.700 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.702+0000 7fea647c8640 1 librados: starting msgr at 2024-03-29T11:27:57.700 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.702+0000 7fea647c8640 1 librados: starting objecter 2024-03-29T11:27:57.700 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.703+0000 7fea647c8640 1 Processor -- start 2024-03-29T11:27:57.700 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.703+0000 7fea647c8640 1 -- start start 2024-03-29T11:27:57.700 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.703+0000 7fea647c8640 1 librados: setting wanted keys 2024-03-29T11:27:57.700 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.703+0000 7fea647c8640 1 librados: calling monclient init 2024-03-29T11:27:57.700 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.703+0000 7fea647c8640 10 monclient: init 2024-03-29T11:27:57.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.703+0000 7fea647c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:57.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.703+0000 7fea647c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:57.706 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.703+0000 7fea647c8640 10 monclient(hunting): picked mon.noname-b con 0x7fea5c06ed30 addr v1:172.21.15.82:6789/0 2024-03-29T11:27:57.707 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea647c8640 10 monclient(hunting): picked mon.noname-a con 0x7fea5c149ae0 addr v1:172.21.15.67:6789/0 2024-03-29T11:27:57.707 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea61d63640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fea5c149ae0 0x7fea5c0a9c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55380/0 (socket says 172.21.15.67:55380) 2024-03-29T11:27:57.707 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea61d63640 1 -- 172.21.15.67:0/2827693997 learned_addr learned my addr 172.21.15.67:0/2827693997 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:27:57.707 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea647c8640 10 monclient(hunting): picked mon.noname-c con 0x7fea5c14a5b0 addr v1:172.21.15.67:6790/0 2024-03-29T11:27:57.707 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea647c8640 1 -- 172.21.15.67:0/2827693997 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5c0ac070 con 0x7fea5c149ae0 2024-03-29T11:27:57.707 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea647c8640 1 -- 172.21.15.67:0/2827693997 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5c1db6a0 con 0x7fea5c06ed30 2024-03-29T11:27:57.707 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea647c8640 1 -- 172.21.15.67:0/2827693997 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5c1dc2d0 con 0x7fea5c14a5b0 2024-03-29T11:27:57.707 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea647c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:57.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea647c8640 10 monclient(hunting): authenticate will time out at 1338.001323s 2024-03-29T11:27:57.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea52ffd640 10 client.?.objecter ms_handle_connect 0x7fea5c14a5b0 2024-03-29T11:27:57.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:57.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea52ffd640 10 client.?.objecter ms_handle_connect 0x7fea5c149ae0 2024-03-29T11:27:57.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:57.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798548174 0 0) 0x7fea5c0ac070 con 0x7fea5c149ae0 2024-03-29T11:27:57.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea52ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:57.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:57.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:57.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.704+0000 7fea52ffd640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:27:57.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea40004eb0 con 0x7fea5c149ae0 2024-03-29T11:27:57.708 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167089981 0 0) 0x7fea5c1dc2d0 con 0x7fea5c14a5b0 2024-03-29T11:27:57.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:57.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:57.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:57.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:27:57.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea5c0ac070 con 0x7fea5c14a5b0 2024-03-29T11:27:57.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2264750518 0 0) 0x7fea5c0ac070 con 0x7fea5c14a5b0 2024-03-29T11:27:57.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea5c1dc2d0 con 0x7fea5c14a5b0 2024-03-29T11:27:57.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fea54001a20 con 0x7fea5c14a5b0 2024-03-29T11:27:57.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:57.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:57.709 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:57.709 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:57.709 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:57.710 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:57.710 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:57.710 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:57.710 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:57.710 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:57.710 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:57.710 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:57.710 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:57.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:57.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1280490128 0 0) 0x7fea5c1dc2d0 con 0x7fea5c14a5b0 2024-03-29T11:27:57.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 >> v1:172.21.15.82:6789/0 conn(0x7fea5c06ed30 legacy=0x7fea5c0aa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:57.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 >> v1:172.21.15.67:6789/0 conn(0x7fea5c149ae0 legacy=0x7fea5c0a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:57.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:57.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 1 monclient: found mon.c 2024-03-29T11:27:57.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:57.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:57.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c1dc460 con 0x7fea5c14a5b0 2024-03-29T11:27:57.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:57.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:57.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.705+0000 7fea52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:57.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea647c8640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:27:57.711 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea647c8640 10 monclient: _renew_subs 2024-03-29T11:27:57.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea647c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:57.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea647c8640 1 -- 172.21.15.67:0/2827693997 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea5c145530 con 0x7fea5c14a5b0 2024-03-29T11:27:57.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea647c8640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:57.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea647c8640 10 monclient: _renew_subs 2024-03-29T11:27:57.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea647c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:57.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea647c8640 1 -- 172.21.15.67:0/2827693997 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea5c13f690 con 0x7fea5c14a5b0 2024-03-29T11:27:57.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea647c8640 1 librados: init done 2024-03-29T11:27:57.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea54001cc0 con 0x7fea5c14a5b0 2024-03-29T11:27:57.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:57.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea60d61640 4 set_mon_vals no callback set 2024-03-29T11:27:57.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fea54004c30 con 0x7fea5c14a5b0 2024-03-29T11:27:57.712 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:57.713 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:57.713 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.706+0000 7fea52ffd640 10 monclient: dump: 2024-03-29T11:27:57.713 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:27:57.713 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:27:57.713 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:57.713 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:27:57.713 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:57.713 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:27:57.713 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:27:57.713 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:27:57.713 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:27:57.713 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:27:57.713 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.707+0000 7fea647c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:57.713 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.707+0000 7fea647c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:57.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.707+0000 7fea647c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:57.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.707+0000 7fea647c8640 1 -- 172.21.15.67:0/2827693997 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea5c06fa70 con 0x7fea5c14a5b0 2024-03-29T11:27:57.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.709+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fea5401ac00 con 0x7fea5c14a5b0 2024-03-29T11:27:57.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.709+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fea54084810 con 0x7fea5c14a5b0 2024-03-29T11:27:57.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.709+0000 7fea52ffd640 10 client.4577.objecter ms_dispatch 0x7fea5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:57.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.709+0000 7fea52ffd640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:57.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.709+0000 7fea52ffd640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:57.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.710+0000 7fea52ffd640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T11:27:57.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.710+0000 7fea62564640 10 monclient: get_auth_request con 0x7fea4006dda0 auth_method 0 2024-03-29T11:27:57.714 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.710+0000 7fea52ffd640 10 client.4577.objecter ms_handle_connect 0x7fea4006dda0 2024-03-29T11:27:57.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.720+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea540500f0 con 0x7fea5c14a5b0 2024-03-29T11:27:57.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.720+0000 7fea52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:57.715 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:57.720+0000 7fea52ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:58.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.088+0000 7fea647c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:58.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.088+0000 7fea647c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:58.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.088+0000 7fea647c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:27:58.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.088+0000 7fea647c8640 1 -- 172.21.15.67:0/2827693997 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea5c1476c0 con 0x7fea5c14a5b0 2024-03-29T11:27:58.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.089+0000 7fea52ffd640 1 -- 172.21.15.67:0/2827693997 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fea54052020 con 0x7fea5c14a5b0 2024-03-29T11:27:58.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.089+0000 7fea52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:58.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.089+0000 7fea52ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:58.079 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:27:58.079 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:58.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.093+0000 7fea647c8640 10 librados: watch_flush enter 2024-03-29T11:27:58.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.093+0000 7fea647c8640 10 librados: watch_flush exit 2024-03-29T11:27:58.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.093+0000 7fea647c8640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T11:27:58.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.093+0000 7fea647c8640 10 client.4577.objecter successfully canceled tick 2024-03-29T11:27:58.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.094+0000 7fea647c8640 1 -- 172.21.15.67:0/2827693997 >> v1:172.21.15.67:6816/404787197 conn(0x7fea4006dda0 legacy=0x7fea40070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.094+0000 7fea647c8640 10 monclient: shutdown 2024-03-29T11:27:58.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.094+0000 7fea647c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:58.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.094+0000 7fea647c8640 1 -- 172.21.15.67:0/2827693997 >> v1:172.21.15.67:6790/0 conn(0x7fea5c14a5b0 legacy=0x7fea5c1dbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.094+0000 7fea647c8640 1 -- 172.21.15.67:0/2827693997 shutdown_connections 2024-03-29T11:27:58.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.094+0000 7fea647c8640 1 -- 172.21.15.67:0/2827693997 >> 172.21.15.67:0/2827693997 conn(0x7fea5c0b1430 msgr2=0x7fea5c071280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:58.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.094+0000 7fea647c8640 1 -- 172.21.15.67:0/2827693997 shutdown_connections 2024-03-29T11:27:58.085 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.095+0000 7fea647c8640 1 -- 172.21.15.67:0/2827693997 wait complete. 2024-03-29T11:27:58.086 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:27:58.095+0000 7fea647c8640 1 librados: shutdown 2024-03-29T11:27:58.100 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:04.909 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:05.066 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.077+0000 7f60128d3640 10 monclient: get_monmap_and_config 2024-03-29T11:28:05.066 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.077+0000 7f60128d3640 10 monclient: build_initial_monmap 2024-03-29T11:28:05.066 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.077+0000 7f60128d3640 10 monclient: monmap: 2024-03-29T11:28:05.066 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:05.066 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:05.066 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:28:05.077869+0000 2024-03-29T11:28:05.066 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:28:05.077869+0000 2024-03-29T11:28:05.066 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:05.067 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:05.067 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:05.067 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:28:05.067 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:28:05.067 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:05.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.080+0000 7f60128d3640 1 Processor -- start 2024-03-29T11:28:05.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.080+0000 7f60128d3640 1 -- start start 2024-03-29T11:28:05.069 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.080+0000 7f60128d3640 10 monclient: init 2024-03-29T11:28:05.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.081+0000 7f60128d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:05.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.081+0000 7f60128d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:05.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.081+0000 7f60128d3640 10 monclient(hunting): picked mon.noname-c con 0x7f600c0b0860 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:05.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.081+0000 7f60128d3640 10 monclient(hunting): picked mon.noname-a con 0x7f600c13ef30 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:05.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.081+0000 7f60128d3640 10 monclient(hunting): picked mon.noname-b con 0x7f600c0af8e0 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:05.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.081+0000 7f60128d3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c0b0c40 con 0x7f600c13ef30 2024-03-29T11:28:05.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.081+0000 7f60128d3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c0aed10 con 0x7f600c0b0860 2024-03-29T11:28:05.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.081+0000 7f60128d3640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c0afd50 con 0x7f600c0af8e0 2024-03-29T11:28:05.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.081+0000 7f60128d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:05.071 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.081+0000 7f60128d3640 10 monclient(hunting): authenticate will time out at 1345.377879s 2024-03-29T11:28:05.073 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.085+0000 7f600b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f600c13ef30 0x7f600c0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35267/0 (socket says 172.21.15.67:35267) 2024-03-29T11:28:05.074 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.085+0000 7f600b7fe640 1 -- 172.21.15.67:0/2127027991 learned_addr learned my addr 172.21.15.67:0/2127027991 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:05.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.085+0000 7f600affd640 1 -- 172.21.15.67:0/2127027991 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870499498 0 0) 0x7f600c0b0c40 con 0x7f600c13ef30 2024-03-29T11:28:05.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.085+0000 7f600affd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:05.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.085+0000 7f600affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:05.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.085+0000 7f600affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:05.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.085+0000 7f600affd640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:28:05.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.085+0000 7f600affd640 1 -- 172.21.15.67:0/2127027991 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff4003440 con 0x7f600c13ef30 2024-03-29T11:28:05.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.086+0000 7f600affd640 1 -- 172.21.15.67:0/2127027991 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 288918084 0 0) 0x7f5ff4003440 con 0x7f600c13ef30 2024-03-29T11:28:05.075 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.086+0000 7f600affd640 1 -- 172.21.15.67:0/2127027991 >> v1:172.21.15.67:6790/0 conn(0x7f600c0af8e0 legacy=0x7f600c149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:05.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.088+0000 7f600affd640 1 -- 172.21.15.67:0/2127027991 >> v1:172.21.15.82:6789/0 conn(0x7f600c0b0860 legacy=0x7f600c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.078 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.088+0000 7f600affd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:05.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.088+0000 7f600affd640 1 monclient: found mon.noname-a 2024-03-29T11:28:05.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.088+0000 7f600affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:05.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.088+0000 7f600affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:28:05.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.088+0000 7f600affd640 1 -- 172.21.15.67:0/2127027991 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c0ab1b0 con 0x7f600c13ef30 2024-03-29T11:28:05.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.088+0000 7f600affd640 10 monclient: _finish_auth 0 2024-03-29T11:28:05.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.088+0000 7f600affd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:05.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.088+0000 7f600affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:05.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.089+0000 7f600affd640 1 -- 172.21.15.67:0/2127027991 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6004001a10 con 0x7f600c13ef30 2024-03-29T11:28:05.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.089+0000 7f60128d3640 5 monclient: authenticate success, global_id 4521 2024-03-29T11:28:05.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.089+0000 7f60128d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:05.079 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.089+0000 7f600affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.089+0000 7f600affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:05.080 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.089+0000 7f600affd640 10 monclient: dump: 2024-03-29T11:28:05.080 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:05.080 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:05.080 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:05.080 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:05.080 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.080 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:05.080 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:05.080 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:05.081 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:05.081 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:05.081 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.089+0000 7f60128d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:05.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.090+0000 7f600affd640 1 -- 172.21.15.67:0/2127027991 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6004001be0 con 0x7f600c13ef30 2024-03-29T11:28:05.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.090+0000 7f600affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:05.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.090+0000 7f600affd640 1 -- 172.21.15.67:0/2127027991 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6004002030 con 0x7f600c13ef30 2024-03-29T11:28:05.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.090+0000 7f60128d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:05.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.090+0000 7f60128d3640 4 set_mon_vals no callback set 2024-03-29T11:28:05.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.091+0000 7f600affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.082 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.091+0000 7f600affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:05.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.091+0000 7f600affd640 10 monclient: dump: 2024-03-29T11:28:05.083 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:05.083 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:05.083 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:05.083 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:05.083 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.083 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:05.083 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:05.083 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:05.083 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:05.083 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:05.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.091+0000 7f60128d3640 10 monclient: shutdown 2024-03-29T11:28:05.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.091+0000 7f60128d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:05.083 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.091+0000 7f60128d3640 1 -- 172.21.15.67:0/2127027991 >> v1:172.21.15.67:6789/0 conn(0x7f600c13ef30 legacy=0x7f600c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.091+0000 7f60128d3640 1 -- 172.21.15.67:0/2127027991 shutdown_connections 2024-03-29T11:28:05.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.092+0000 7f60128d3640 1 -- 172.21.15.67:0/2127027991 >> 172.21.15.67:0/2127027991 conn(0x7f600c06b660 msgr2=0x7f600c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:05.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.093+0000 7f60128d3640 1 -- 172.21.15.67:0/2127027991 shutdown_connections 2024-03-29T11:28:05.084 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.094+0000 7f60128d3640 1 -- 172.21.15.67:0/2127027991 wait complete. 2024-03-29T11:28:05.087 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.095+0000 7f60128d3640 10 monclient: build_initial_monmap 2024-03-29T11:28:05.088 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.095+0000 7f60128d3640 10 monclient: monmap: 2024-03-29T11:28:05.088 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:05.088 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:05.088 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:28:05.088 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:28:05.088 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:05.088 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:05.088 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:05.088 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:28:05.089 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:28:05.089 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:05.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.095+0000 7f60128d3640 1 librados: starting msgr at 2024-03-29T11:28:05.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.095+0000 7f60128d3640 1 librados: starting objecter 2024-03-29T11:28:05.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.096+0000 7f60128d3640 1 Processor -- start 2024-03-29T11:28:05.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.098+0000 7f60128d3640 1 -- start start 2024-03-29T11:28:05.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.098+0000 7f60128d3640 1 librados: setting wanted keys 2024-03-29T11:28:05.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.098+0000 7f60128d3640 1 librados: calling monclient init 2024-03-29T11:28:05.089 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.098+0000 7f60128d3640 10 monclient: init 2024-03-29T11:28:05.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.099+0000 7f60128d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:05.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.100+0000 7f60128d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:05.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.100+0000 7f60128d3640 10 monclient(hunting): picked mon.noname-b con 0x7f600c0af8e0 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:05.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.100+0000 7f60128d3640 10 monclient(hunting): picked mon.noname-c con 0x7f600c0b0860 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:05.090 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.100+0000 7f60128d3640 10 monclient(hunting): picked mon.noname-a con 0x7f600c13ef30 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:05.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.100+0000 7f60128d3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c1d6b50 con 0x7f600c13ef30 2024-03-29T11:28:05.092 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.100+0000 7f60128d3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c133150 con 0x7f600c0af8e0 2024-03-29T11:28:05.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.100+0000 7f60128d3640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c1d6fe0 con 0x7f600c0b0860 2024-03-29T11:28:05.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.100+0000 7f60128d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:05.093 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.100+0000 7f60128d3640 10 monclient(hunting): authenticate will time out at 1345.396805s 2024-03-29T11:28:05.094 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.104+0000 7f600b7fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f600c0b0860 0x7f600c132330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:57835/0 (socket says 172.21.15.67:57835) 2024-03-29T11:28:05.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.104+0000 7f600b7fe640 1 -- 172.21.15.67:0/2031946003 learned_addr learned my addr 172.21.15.67:0/2031946003 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:05.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.105+0000 7f6008ff9640 10 client.?.objecter ms_handle_connect 0x7f600c0b0860 2024-03-29T11:28:05.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.105+0000 7f6008ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:05.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.105+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824978389 0 0) 0x7f600c1d6fe0 con 0x7f600c0b0860 2024-03-29T11:28:05.095 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.105+0000 7f6008ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:05.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.105+0000 7f6008ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:05.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.105+0000 7f6008ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:05.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.105+0000 7f6008ff9640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:28:05.096 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.105+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff8004eb0 con 0x7f600c0b0860 2024-03-29T11:28:05.103 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.106+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1498792239 0 0) 0x7f5ff8004eb0 con 0x7f600c0b0860 2024-03-29T11:28:05.103 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.106+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f600c1d6fe0 con 0x7f600c0b0860 2024-03-29T11:28:05.103 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.106+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6004001f30 con 0x7f600c0b0860 2024-03-29T11:28:05.103 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.107+0000 7f6008ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.103 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.107+0000 7f6008ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:05.104 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.107+0000 7f6008ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:05.104 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:05.104 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:05.104 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:05.104 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:05.104 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.104 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:05.104 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:05.104 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:05.104 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:05.104 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:05.104 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.107+0000 7f6008ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:05.104 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.107+0000 7f6008ff9640 10 client.?.objecter ms_handle_connect 0x7f600c0af8e0 2024-03-29T11:28:05.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.107+0000 7f6008ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:05.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.107+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492807324 0 0) 0x7f600c133150 con 0x7f600c0af8e0 2024-03-29T11:28:05.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.107+0000 7f6008ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:05.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.107+0000 7f6008ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:05.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.107+0000 7f6008ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:05.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.107+0000 7f6008ff9640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:28:05.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.107+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff8004eb0 con 0x7f600c0af8e0 2024-03-29T11:28:05.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.108+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 333781661 0 0) 0x7f5ff8004eb0 con 0x7f600c0af8e0 2024-03-29T11:28:05.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.108+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f600c133150 con 0x7f600c0af8e0 2024-03-29T11:28:05.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.108+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5ffc0018b0 con 0x7f600c0af8e0 2024-03-29T11:28:05.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.108+0000 7f6008ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.108+0000 7f6008ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:05.105 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.108+0000 7f6008ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:05.105 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:05.106 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:05.106 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:05.106 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:05.106 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.106 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:05.106 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:05.106 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:05.106 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:05.106 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:05.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.108+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 157191157 0 0) 0x7f600c1d6fe0 con 0x7f600c0b0860 2024-03-29T11:28:05.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.108+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 >> v1:172.21.15.82:6789/0 conn(0x7f600c0af8e0 legacy=0x7f600c137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.108+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 >> v1:172.21.15.67:6789/0 conn(0x7f600c13ef30 legacy=0x7f600c132a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.106 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.109+0000 7f6008ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:28:05.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.109+0000 7f6008ff9640 1 monclient: found mon.c 2024-03-29T11:28:05.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.109+0000 7f6008ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:05.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.109+0000 7f6008ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:05.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.109+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c071960 con 0x7f600c0b0860 2024-03-29T11:28:05.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.109+0000 7f6008ff9640 10 monclient: _finish_auth 0 2024-03-29T11:28:05.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.109+0000 7f6008ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:28:05.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.109+0000 7f6008ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:05.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.109+0000 7f60128d3640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:28:05.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.109+0000 7f60128d3640 10 monclient: _renew_subs 2024-03-29T11:28:05.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.109+0000 7f60128d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:05.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.109+0000 7f60128d3640 1 -- 172.21.15.67:0/2031946003 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600c071b50 con 0x7f600c0b0860 2024-03-29T11:28:05.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.109+0000 7f60128d3640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:05.107 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.109+0000 7f60128d3640 10 monclient: _renew_subs 2024-03-29T11:28:05.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.110+0000 7f60128d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:05.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.110+0000 7f60128d3640 1 -- 172.21.15.67:0/2031946003 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f600c138070 con 0x7f600c0b0860 2024-03-29T11:28:05.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.110+0000 7f60128d3640 1 librados: init done 2024-03-29T11:28:05.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.111+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6004002090 con 0x7f600c0b0860 2024-03-29T11:28:05.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.111+0000 7f6008ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:05.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.111+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f6004005030 con 0x7f600c0b0860 2024-03-29T11:28:05.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.111+0000 7f6008ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.111+0000 7f6008ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:05.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.111+0000 7f600a7fc640 4 set_mon_vals no callback set 2024-03-29T11:28:05.108 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.111+0000 7f6008ff9640 10 monclient: dump: 2024-03-29T11:28:05.108 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:05.108 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:05.108 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:05.108 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:05.109 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.109 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:05.109 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:05.109 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:05.109 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:05.109 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:05.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.112+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f600401b0c0 con 0x7f600c0b0860 2024-03-29T11:28:05.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.113+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f6004084ed0 con 0x7f600c0b0860 2024-03-29T11:28:05.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.113+0000 7f6008ff9640 10 client.4580.objecter ms_dispatch 0x7f600c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:05.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.113+0000 7f6008ff9640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:05.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.113+0000 7f6008ff9640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:05.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.113+0000 7f6008ff9640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T11:28:05.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.113+0000 7f600bfff640 10 monclient: get_auth_request con 0x7f5ff80700b0 auth_method 0 2024-03-29T11:28:05.109 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.114+0000 7f60128d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:05.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.114+0000 7f60128d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:05.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.114+0000 7f60128d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:05.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.114+0000 7f60128d3640 1 -- 172.21.15.67:0/2031946003 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fec005080 con 0x7f600c0b0860 2024-03-29T11:28:05.110 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.114+0000 7f6008ff9640 10 client.4580.objecter ms_handle_connect 0x7f5ff80700b0 2024-03-29T11:28:05.114 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.125+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f60040507b0 con 0x7f600c0b0860 2024-03-29T11:28:05.114 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.125+0000 7f6008ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:05.114 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.125+0000 7f6008ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.500+0000 7f60128d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.501+0000 7f60128d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.501+0000 7f60128d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.501+0000 7f60128d3640 1 -- 172.21.15.67:0/2031946003 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fec005330 con 0x7f600c0b0860 2024-03-29T11:28:05.490 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.501+0000 7f6008ff9640 1 -- 172.21.15.67:0/2031946003 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f60040526e0 con 0x7f600c0b0860 2024-03-29T11:28:05.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.501+0000 7f6008ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:05.491 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.501+0000 7f6008ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:05.491 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:28:05.491 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:05.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.505+0000 7f60128d3640 10 librados: watch_flush enter 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.505+0000 7f60128d3640 10 librados: watch_flush exit 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.505+0000 7f60128d3640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.505+0000 7f60128d3640 10 client.4580.objecter successfully canceled tick 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.505+0000 7f60128d3640 1 -- 172.21.15.67:0/2031946003 >> v1:172.21.15.67:6816/404787197 conn(0x7f5ff80700b0 legacy=0x7f5ff8072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.505+0000 7f60128d3640 10 monclient: shutdown 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.505+0000 7f60128d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.505+0000 7f60128d3640 1 -- 172.21.15.67:0/2031946003 >> v1:172.21.15.67:6790/0 conn(0x7f600c0b0860 legacy=0x7f600c132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.505+0000 7f60128d3640 1 -- 172.21.15.67:0/2031946003 shutdown_connections 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.505+0000 7f60128d3640 1 -- 172.21.15.67:0/2031946003 >> 172.21.15.67:0/2031946003 conn(0x7f600c06b660 msgr2=0x7f600c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.505+0000 7f60128d3640 1 -- 172.21.15.67:0/2031946003 shutdown_connections 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.506+0000 7f60128d3640 1 -- 172.21.15.67:0/2031946003 wait complete. 2024-03-29T11:28:05.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:05.506+0000 7f60128d3640 1 librados: shutdown 2024-03-29T11:28:05.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:12.316 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:12.468 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.479+0000 7efeb759e640 10 monclient: get_monmap_and_config 2024-03-29T11:28:12.469 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.479+0000 7efeb759e640 10 monclient: build_initial_monmap 2024-03-29T11:28:12.469 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.479+0000 7efeb759e640 10 monclient: monmap: 2024-03-29T11:28:12.469 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:12.469 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:12.469 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:28:12.480418+0000 2024-03-29T11:28:12.469 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:28:12.480418+0000 2024-03-29T11:28:12.469 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:12.470 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:12.470 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:12.470 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:28:12.470 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:28:12.470 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:12.470 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.482+0000 7efeb759e640 1 Processor -- start 2024-03-29T11:28:12.472 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.482+0000 7efeb759e640 1 -- start start 2024-03-29T11:28:12.472 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.482+0000 7efeb759e640 10 monclient: init 2024-03-29T11:28:12.472 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.483+0000 7efeb759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:12.472 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.483+0000 7efeb759e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:12.472 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.483+0000 7efeb759e640 10 monclient(hunting): picked mon.noname-c con 0x7efeb0006e30 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:12.472 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.483+0000 7efeb759e640 10 monclient(hunting): picked mon.noname-b con 0x7efeb0035e00 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:12.472 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.483+0000 7efeb759e640 10 monclient(hunting): picked mon.noname-a con 0x7efeb00c9670 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:12.472 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.483+0000 7efeb759e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb0035970 con 0x7efeb00c9670 2024-03-29T11:28:12.472 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.483+0000 7efeb759e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb0036280 con 0x7efeb0006e30 2024-03-29T11:28:12.472 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.483+0000 7efeb759e640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb0035640 con 0x7efeb0035e00 2024-03-29T11:28:12.473 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.483+0000 7efeb759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:12.473 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.483+0000 7efeb759e640 10 monclient(hunting): authenticate will time out at 1352.779128s 2024-03-29T11:28:12.474 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.485+0000 7efeb6d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efeb00c9670 0x7efeb00d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59440/0 (socket says 172.21.15.67:59440) 2024-03-29T11:28:12.474 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.485+0000 7efeb6d9d640 1 -- 172.21.15.67:0/207550621 learned_addr learned my addr 172.21.15.67:0/207550621 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:12.476 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.485+0000 7efeb559a640 1 -- 172.21.15.67:0/207550621 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806484402 0 0) 0x7efeb0035970 con 0x7efeb00c9670 2024-03-29T11:28:12.476 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.485+0000 7efeb559a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:12.476 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.485+0000 7efeb559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:12.476 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.485+0000 7efeb559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:12.477 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.485+0000 7efeb559a640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:28:12.477 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.485+0000 7efeb559a640 1 -- 172.21.15.67:0/207550621 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea0003520 con 0x7efeb00c9670 2024-03-29T11:28:12.477 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.485+0000 7efeb559a640 1 -- 172.21.15.67:0/207550621 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4033375141 0 0) 0x7efea0003520 con 0x7efeb00c9670 2024-03-29T11:28:12.477 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.485+0000 7efeb559a640 1 -- 172.21.15.67:0/207550621 >> v1:172.21.15.67:6790/0 conn(0x7efeb0035e00 legacy=0x7efeb00c8f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:12.477 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 1 -- 172.21.15.67:0/207550621 >> v1:172.21.15.82:6789/0 conn(0x7efeb0006e30 legacy=0x7efeb00cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:12.477 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 10 monclient: _finish_hunting 0 2024-03-29T11:28:12.477 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 1 monclient: found mon.noname-a 2024-03-29T11:28:12.477 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:12.477 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:28:12.477 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 1 -- 172.21.15.67:0/207550621 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb0034e20 con 0x7efeb00c9670 2024-03-29T11:28:12.477 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 10 monclient: _finish_auth 0 2024-03-29T11:28:12.477 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 10 monclient: _check_auth_tickets 2024-03-29T11:28:12.478 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:12.478 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb759e640 5 monclient: authenticate success, global_id 4524 2024-03-29T11:28:12.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:12.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 1 -- 172.21.15.67:0/207550621 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7efeb804f730 con 0x7efeb00c9670 2024-03-29T11:28:12.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:12.480 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 10 monclient: dump: 2024-03-29T11:28:12.480 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:12.480 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:12.480 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:12.481 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:12.481 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.481 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:12.481 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:12.481 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:12.481 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:12.481 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:12.481 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:12.481 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 1 -- 172.21.15.67:0/207550621 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeb80008d0 con 0x7efeb00c9670 2024-03-29T11:28:12.481 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:12.481 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:12.482 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.487+0000 7efeb759e640 4 set_mon_vals no callback set 2024-03-29T11:28:12.482 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.488+0000 7efeb759e640 10 monclient: shutdown 2024-03-29T11:28:12.482 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.488+0000 7efeb759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:12.482 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.488+0000 7efeb759e640 1 -- 172.21.15.67:0/207550621 >> v1:172.21.15.67:6789/0 conn(0x7efeb00c9670 legacy=0x7efeb00d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.482 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.489+0000 7efeb759e640 1 -- 172.21.15.67:0/207550621 shutdown_connections 2024-03-29T11:28:12.482 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.489+0000 7efeb759e640 1 -- 172.21.15.67:0/207550621 >> 172.21.15.67:0/207550621 conn(0x7efeb0005c00 msgr2=0x7efeb0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:12.482 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.489+0000 7efeb759e640 1 -- 172.21.15.67:0/207550621 shutdown_connections 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.489+0000 7efeb759e640 1 -- 172.21.15.67:0/207550621 wait complete. 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.490+0000 7efeb759e640 10 monclient: build_initial_monmap 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.490+0000 7efeb759e640 10 monclient: monmap: 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.490+0000 7efeb759e640 1 librados: starting msgr at 2024-03-29T11:28:12.483 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.490+0000 7efeb759e640 1 librados: starting objecter 2024-03-29T11:28:12.484 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.491+0000 7efeb759e640 1 Processor -- start 2024-03-29T11:28:12.484 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.491+0000 7efeb759e640 1 -- start start 2024-03-29T11:28:12.484 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.491+0000 7efeb759e640 1 librados: setting wanted keys 2024-03-29T11:28:12.484 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.491+0000 7efeb759e640 1 librados: calling monclient init 2024-03-29T11:28:12.484 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.491+0000 7efeb759e640 10 monclient: init 2024-03-29T11:28:12.492 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.492+0000 7efeb759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:12.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.492+0000 7efeb759e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:12.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.492+0000 7efeb759e640 10 monclient(hunting): picked mon.noname-b con 0x7efeb0006e30 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:12.493 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.492+0000 7efeb759e640 10 monclient(hunting): picked mon.noname-a con 0x7efeb0035e00 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:12.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.494+0000 7efeb759e640 10 monclient(hunting): picked mon.noname-c con 0x7efeb00c9670 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:12.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.494+0000 7efeb759e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb002db20 con 0x7efeb0035e00 2024-03-29T11:28:12.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.494+0000 7efeb759e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb002dd70 con 0x7efeb0006e30 2024-03-29T11:28:12.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.494+0000 7efeb759e640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb002e050 con 0x7efeb00c9670 2024-03-29T11:28:12.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.494+0000 7efeb759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:12.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.494+0000 7efeb759e640 10 monclient(hunting): authenticate will time out at 1352.790642s 2024-03-29T11:28:12.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.496+0000 7efeb6d9d640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7efeb00c9670 0x7efeb00c0be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:60027/0 (socket says 172.21.15.67:60027) 2024-03-29T11:28:12.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.496+0000 7efeb6d9d640 1 -- 172.21.15.67:0/684730098 learned_addr learned my addr 172.21.15.67:0/684730098 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:12.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.496+0000 7efea6ffd640 10 client.?.objecter ms_handle_connect 0x7efeb00c9670 2024-03-29T11:28:12.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.496+0000 7efea6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:12.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537001609 0 0) 0x7efeb002e050 con 0x7efeb00c9670 2024-03-29T11:28:12.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:12.494 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:12.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:12.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:28:12.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe98004eb0 con 0x7efeb00c9670 2024-03-29T11:28:12.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 10 client.?.objecter ms_handle_connect 0x7efeb0035e00 2024-03-29T11:28:12.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:12.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 10 client.?.objecter ms_handle_connect 0x7efeb0006e30 2024-03-29T11:28:12.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:12.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3915457742 0 0) 0x7efe98004eb0 con 0x7efeb00c9670 2024-03-29T11:28:12.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efeb002e050 con 0x7efeb00c9670 2024-03-29T11:28:12.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7efeb804f400 con 0x7efeb00c9670 2024-03-29T11:28:12.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.495 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:12.496 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:12.496 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:12.496 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:12.496 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:12.496 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:12.496 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.496 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:12.496 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:12.496 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:12.496 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:12.496 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:12.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.497+0000 7efea6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:12.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.498+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2769921585 0 0) 0x7efeb002e050 con 0x7efeb00c9670 2024-03-29T11:28:12.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.498+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 >> v1:172.21.15.82:6789/0 conn(0x7efeb0006e30 legacy=0x7efeb00338d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.498+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 >> v1:172.21.15.67:6789/0 conn(0x7efeb0035e00 legacy=0x7efeb01641f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.498+0000 7efea6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:12.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.498+0000 7efea6ffd640 1 monclient: found mon.c 2024-03-29T11:28:12.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.498+0000 7efea6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:12.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efea6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:12.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb00330d0 con 0x7efeb00c9670 2024-03-29T11:28:12.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efea6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:12.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efea6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:12.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:12.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efeb759e640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:28:12.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efeb759e640 10 monclient: _renew_subs 2024-03-29T11:28:12.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efeb759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:12.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efeb759e640 1 -- 172.21.15.67:0/684730098 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb00333b0 con 0x7efeb00c9670 2024-03-29T11:28:12.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efeb759e640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:12.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efeb759e640 10 monclient: _renew_subs 2024-03-29T11:28:12.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efeb759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:12.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efeb759e640 1 -- 172.21.15.67:0/684730098 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb00cb350 con 0x7efeb00c9670 2024-03-29T11:28:12.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efeb759e640 1 librados: init done 2024-03-29T11:28:12.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeb804fdd0 con 0x7efeb00c9670 2024-03-29T11:28:12.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efea6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:12.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7efeb8075710 con 0x7efeb00c9670 2024-03-29T11:28:12.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efea6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efea6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:12.499 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efea6ffd640 10 monclient: dump: 2024-03-29T11:28:12.499 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:12.499 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:12.499 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:12.499 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:12.499 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.499 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:12.499 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:12.499 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:12.499 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:12.500 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:12.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.499+0000 7efeb4d99640 4 set_mon_vals no callback set 2024-03-29T11:28:12.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.501+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7efeb8067ea0 con 0x7efeb00c9670 2024-03-29T11:28:12.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.501+0000 7efeb759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:12.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.501+0000 7efeb759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:12.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.501+0000 7efeb759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:12.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.501+0000 7efeb759e640 1 -- 172.21.15.67:0/684730098 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c005080 con 0x7efeb00c9670 2024-03-29T11:28:12.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.502+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7efeb80ff650 con 0x7efeb00c9670 2024-03-29T11:28:12.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.502+0000 7efeb659c640 10 monclient: get_auth_request con 0x7efe9806dac0 auth_method 0 2024-03-29T11:28:12.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.503+0000 7efea6ffd640 10 client.4583.objecter ms_dispatch 0x7efeb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:12.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.503+0000 7efea6ffd640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:12.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.503+0000 7efea6ffd640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:12.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.503+0000 7efea6ffd640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-29T11:28:12.500 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.503+0000 7efea6ffd640 10 client.4583.objecter ms_handle_connect 0x7efe9806dac0 2024-03-29T11:28:12.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.509+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efeb80caf30 con 0x7efeb00c9670 2024-03-29T11:28:12.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.509+0000 7efea6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:12.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.510+0000 7efea6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:12.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.848+0000 7efeb759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:12.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.848+0000 7efeb759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:12.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.848+0000 7efeb759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:12.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.848+0000 7efeb759e640 1 -- 172.21.15.67:0/684730098 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe9c005330 con 0x7efeb00c9670 2024-03-29T11:28:12.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.849+0000 7efea6ffd640 1 -- 172.21.15.67:0/684730098 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7efeb80cce60 con 0x7efeb00c9670 2024-03-29T11:28:12.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.849+0000 7efea6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:12.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.849+0000 7efea6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:12.839 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:28:12.840 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:12.843 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.854+0000 7efeb759e640 10 librados: watch_flush enter 2024-03-29T11:28:12.845 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.856+0000 7efeb759e640 10 librados: watch_flush exit 2024-03-29T11:28:12.845 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.856+0000 7efeb759e640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-29T11:28:12.845 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.856+0000 7efeb759e640 10 client.4583.objecter successfully canceled tick 2024-03-29T11:28:12.845 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.856+0000 7efeb759e640 1 -- 172.21.15.67:0/684730098 >> v1:172.21.15.67:6816/404787197 conn(0x7efe9806dac0 legacy=0x7efe9806ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.845 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.856+0000 7efeb759e640 10 monclient: shutdown 2024-03-29T11:28:12.846 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.856+0000 7efeb759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:12.846 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.856+0000 7efeb759e640 1 -- 172.21.15.67:0/684730098 >> v1:172.21.15.67:6790/0 conn(0x7efeb00c9670 legacy=0x7efeb00c0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.858+0000 7efeb759e640 1 -- 172.21.15.67:0/684730098 shutdown_connections 2024-03-29T11:28:12.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.858+0000 7efeb759e640 1 -- 172.21.15.67:0/684730098 >> 172.21.15.67:0/684730098 conn(0x7efeb0005c00 msgr2=0x7efeb003ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:12.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.858+0000 7efeb759e640 1 -- 172.21.15.67:0/684730098 shutdown_connections 2024-03-29T11:28:12.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.858+0000 7efeb759e640 1 -- 172.21.15.67:0/684730098 wait complete. 2024-03-29T11:28:12.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:12.858+0000 7efeb759e640 1 librados: shutdown 2024-03-29T11:28:12.866 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:19.674 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:19.821 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.831+0000 7f70425ed640 10 monclient: get_monmap_and_config 2024-03-29T11:28:19.821 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.831+0000 7f70425ed640 10 monclient: build_initial_monmap 2024-03-29T11:28:19.821 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.831+0000 7f70425ed640 10 monclient: monmap: 2024-03-29T11:28:19.821 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:19.821 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:19.821 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:28:19.832554+0000 2024-03-29T11:28:19.821 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:28:19.832554+0000 2024-03-29T11:28:19.821 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:19.822 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:19.822 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:19.822 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:28:19.822 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:28:19.822 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:19.822 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.832+0000 7f70425ed640 1 Processor -- start 2024-03-29T11:28:19.822 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.832+0000 7f70425ed640 1 -- start start 2024-03-29T11:28:19.822 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.832+0000 7f70425ed640 10 monclient: init 2024-03-29T11:28:19.825 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.833+0000 7f70425ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:19.825 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.834+0000 7f70425ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:19.825 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.834+0000 7f70425ed640 10 monclient(hunting): picked mon.noname-a con 0x7f703c1457f0 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:19.825 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.834+0000 7f70425ed640 10 monclient(hunting): picked mon.noname-b con 0x7f703c13f650 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:19.825 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.834+0000 7f70425ed640 10 monclient(hunting): picked mon.noname-c con 0x7f703c0a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:19.825 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.834+0000 7f70425ed640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c0aaef0 con 0x7f703c1457f0 2024-03-29T11:28:19.825 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.834+0000 7f70425ed640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c145bd0 con 0x7f703c0a9d70 2024-03-29T11:28:19.825 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.834+0000 7f70425ed640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c145eb0 con 0x7f703c13f650 2024-03-29T11:28:19.825 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.834+0000 7f70425ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:19.826 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.834+0000 7f70425ed640 10 monclient(hunting): authenticate will time out at 1360.130010s 2024-03-29T11:28:19.826 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.834+0000 7f703bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f703c1457f0 0x7f703c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41710/0 (socket says 172.21.15.67:41710) 2024-03-29T11:28:19.826 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.834+0000 7f703bfff640 1 -- 172.21.15.67:0/1252255730 learned_addr learned my addr 172.21.15.67:0/1252255730 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:19.826 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.834+0000 7f703affd640 1 -- 172.21.15.67:0/1252255730 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534753856 0 0) 0x7f703c0aaef0 con 0x7f703c1457f0 2024-03-29T11:28:19.826 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.834+0000 7f703affd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:19.826 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.835+0000 7f703affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:19.826 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.835+0000 7f703affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:19.826 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.835+0000 7f703affd640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:28:19.826 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.835+0000 7f703affd640 1 -- 172.21.15.67:0/1252255730 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7024003520 con 0x7f703c1457f0 2024-03-29T11:28:19.826 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.835+0000 7f703affd640 1 -- 172.21.15.67:0/1252255730 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2943483318 0 0) 0x7f7024003520 con 0x7f703c1457f0 2024-03-29T11:28:19.826 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.835+0000 7f703affd640 1 -- 172.21.15.67:0/1252255730 >> v1:172.21.15.67:6790/0 conn(0x7f703c13f650 legacy=0x7f703c0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:19.827 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.837+0000 7f703affd640 1 -- 172.21.15.67:0/1252255730 >> v1:172.21.15.82:6789/0 conn(0x7f703c0a9d70 legacy=0x7f703c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:19.829 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.838+0000 7f703affd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:19.829 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.838+0000 7f703affd640 1 monclient: found mon.noname-a 2024-03-29T11:28:19.829 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.838+0000 7f703affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:19.829 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.838+0000 7f703affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:28:19.829 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.838+0000 7f703affd640 1 -- 172.21.15.67:0/1252255730 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c146340 con 0x7f703c1457f0 2024-03-29T11:28:19.829 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.838+0000 7f703affd640 10 monclient: _finish_auth 0 2024-03-29T11:28:19.830 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.838+0000 7f703affd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:19.830 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.838+0000 7f703affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:19.830 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.838+0000 7f703affd640 1 -- 172.21.15.67:0/1252255730 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7030001690 con 0x7f703c1457f0 2024-03-29T11:28:19.830 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.838+0000 7f703affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:19.830 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.838+0000 7f703affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:19.830 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.838+0000 7f703affd640 10 monclient: dump: 2024-03-29T11:28:19.830 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:19.830 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:19.830 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:19.830 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:19.830 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:19.830 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:19.830 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:19.831 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:19.831 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:19.831 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:19.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.838+0000 7f70425ed640 5 monclient: authenticate success, global_id 4527 2024-03-29T11:28:19.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.838+0000 7f70425ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:19.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.839+0000 7f703affd640 1 -- 172.21.15.67:0/1252255730 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7030001860 con 0x7f703c1457f0 2024-03-29T11:28:19.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.839+0000 7f703affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:19.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.839+0000 7f70425ed640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:19.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.839+0000 7f70425ed640 4 set_mon_vals no callback set 2024-03-29T11:28:19.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.840+0000 7f70425ed640 10 monclient: shutdown 2024-03-29T11:28:19.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.840+0000 7f70425ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:19.831 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.840+0000 7f70425ed640 1 -- 172.21.15.67:0/1252255730 >> v1:172.21.15.67:6789/0 conn(0x7f703c1457f0 legacy=0x7f703c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:19.832 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.842+0000 7f70425ed640 1 -- 172.21.15.67:0/1252255730 shutdown_connections 2024-03-29T11:28:19.832 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.842+0000 7f70425ed640 1 -- 172.21.15.67:0/1252255730 >> 172.21.15.67:0/1252255730 conn(0x7f703c066a00 msgr2=0x7f703c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:19.832 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.843+0000 7f70425ed640 1 -- 172.21.15.67:0/1252255730 shutdown_connections 2024-03-29T11:28:19.833 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.844+0000 7f70425ed640 1 -- 172.21.15.67:0/1252255730 wait complete. 2024-03-29T11:28:19.834 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.845+0000 7f70425ed640 10 monclient: build_initial_monmap 2024-03-29T11:28:19.834 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.845+0000 7f70425ed640 10 monclient: monmap: 2024-03-29T11:28:19.834 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:19.834 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:19.834 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:28:19.835 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:28:19.835 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:19.835 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:19.835 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:19.835 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:28:19.835 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:28:19.835 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:19.836 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.846+0000 7f70425ed640 1 librados: starting msgr at 2024-03-29T11:28:19.836 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.846+0000 7f70425ed640 1 librados: starting objecter 2024-03-29T11:28:19.836 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.846+0000 7f70425ed640 1 Processor -- start 2024-03-29T11:28:19.836 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.846+0000 7f70425ed640 1 -- start start 2024-03-29T11:28:19.836 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.846+0000 7f70425ed640 1 librados: setting wanted keys 2024-03-29T11:28:19.836 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.846+0000 7f70425ed640 1 librados: calling monclient init 2024-03-29T11:28:19.836 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.846+0000 7f70425ed640 10 monclient: init 2024-03-29T11:28:19.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.847+0000 7f70425ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:19.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.847+0000 7f70425ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:19.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.847+0000 7f70425ed640 10 monclient(hunting): picked mon.noname-c con 0x7f703c0a9d70 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:19.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.848+0000 7f70425ed640 10 monclient(hunting): picked mon.noname-b con 0x7f703c13f650 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:19.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.848+0000 7f70425ed640 10 monclient(hunting): picked mon.noname-a con 0x7f703c1457f0 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:19.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.848+0000 7f7040b8a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f703c1457f0 0x7f703c1da550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46416/0 (socket says 172.21.15.67:46416) 2024-03-29T11:28:19.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.848+0000 7f7040b8a640 1 -- 172.21.15.67:0/304562783 learned_addr learned my addr 172.21.15.67:0/304562783 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:19.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.848+0000 7f70425ed640 1 -- 172.21.15.67:0/304562783 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c13b080 con 0x7f703c1457f0 2024-03-29T11:28:19.838 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.848+0000 7f7038ff9640 10 client.?.objecter ms_handle_connect 0x7f703c1457f0 2024-03-29T11:28:19.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.848+0000 7f7038ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:19.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.848+0000 7f70425ed640 1 -- 172.21.15.67:0/304562783 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c13ad50 con 0x7f703c13f650 2024-03-29T11:28:19.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.848+0000 7f70425ed640 1 -- 172.21.15.67:0/304562783 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c1397d0 con 0x7f703c0a9d70 2024-03-29T11:28:19.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.848+0000 7f70425ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:19.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.849+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138196711 0 0) 0x7f703c13b080 con 0x7f703c1457f0 2024-03-29T11:28:19.839 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.849+0000 7f70425ed640 10 monclient(hunting): authenticate will time out at 1360.144994s 2024-03-29T11:28:19.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.852+0000 7f7038ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:19.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.852+0000 7f7038ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:19.841 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.852+0000 7f7038ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:19.842 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.852+0000 7f7038ff9640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:28:19.842 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.852+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7028004eb0 con 0x7f703c1457f0 2024-03-29T11:28:19.843 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.852+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4287062965 0 0) 0x7f7028004eb0 con 0x7f703c1457f0 2024-03-29T11:28:19.843 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.852+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f703c13b080 con 0x7f703c1457f0 2024-03-29T11:28:19.843 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.852+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7034001c90 con 0x7f703c1457f0 2024-03-29T11:28:19.843 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.852+0000 7f7038ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:19.843 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.852+0000 7f7038ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:19.843 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.852+0000 7f7038ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:19.843 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:19.843 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:19.843 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:19.843 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:19.843 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:19.843 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:19.844 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:19.844 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:19.844 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:19.844 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:19.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.853+0000 7f7038ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:19.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.853+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 81525738 0 0) 0x7f703c13b080 con 0x7f703c1457f0 2024-03-29T11:28:19.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.853+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 >> v1:172.21.15.67:6790/0 conn(0x7f703c0a9d70 legacy=0x7f703c142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:19.844 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.853+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 >> v1:172.21.15.82:6789/0 conn(0x7f703c13f650 legacy=0x7f703c138f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:19.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.855+0000 7f7038ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:28:19.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.855+0000 7f7038ff9640 1 monclient: found mon.a 2024-03-29T11:28:19.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.855+0000 7f7038ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:19.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.855+0000 7f7038ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:28:19.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.855+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c13b510 con 0x7f703c1457f0 2024-03-29T11:28:19.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.855+0000 7f7038ff9640 10 monclient: _finish_auth 0 2024-03-29T11:28:19.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.855+0000 7f7038ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:28:19.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.855+0000 7f7038ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:19.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7034003990 con 0x7f703c1457f0 2024-03-29T11:28:19.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f7038ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:19.847 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7034003de0 con 0x7f703c1457f0 2024-03-29T11:28:19.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f7038ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:19.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f7038ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:19.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f703a7fc640 4 set_mon_vals no callback set 2024-03-29T11:28:19.848 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f7038ff9640 10 monclient: dump: 2024-03-29T11:28:19.848 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:19.848 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:19.848 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:19.848 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:19.848 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:19.848 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:19.848 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:19.848 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:19.848 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:19.848 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:19.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f70425ed640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:28:19.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f70425ed640 10 monclient: _renew_subs 2024-03-29T11:28:19.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f70425ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:28:19.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f70425ed640 1 -- 172.21.15.67:0/304562783 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703c13b700 con 0x7f703c1457f0 2024-03-29T11:28:19.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f70425ed640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:19.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f70425ed640 10 monclient: _renew_subs 2024-03-29T11:28:19.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f70425ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:28:19.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f70425ed640 1 -- 172.21.15.67:0/304562783 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f703c1d45b0 con 0x7f703c1457f0 2024-03-29T11:28:19.849 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.856+0000 7f70425ed640 1 librados: init done 2024-03-29T11:28:19.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.858+0000 7f70425ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:19.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.858+0000 7f70425ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:19.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.858+0000 7f70425ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:28:19.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.858+0000 7f70425ed640 1 -- 172.21.15.67:0/304562783 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703c139960 con 0x7f703c1457f0 2024-03-29T11:28:19.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.859+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f7034003fc0 con 0x7f703c1457f0 2024-03-29T11:28:19.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.860+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f7034084e90 con 0x7f703c1457f0 2024-03-29T11:28:19.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.860+0000 7f7038ff9640 10 client.4530.objecter ms_dispatch 0x7f703c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:19.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.860+0000 7f7038ff9640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:19.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.860+0000 7f7038ff9640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:19.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.860+0000 7f7038ff9640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-29T11:28:19.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.860+0000 7f703bfff640 10 monclient: get_auth_request con 0x7f702806dc50 auth_method 0 2024-03-29T11:28:19.851 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.860+0000 7f7038ff9640 10 client.4530.objecter ms_handle_connect 0x7f702806dc50 2024-03-29T11:28:19.859 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.870+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7034050770 con 0x7f703c1457f0 2024-03-29T11:28:19.859 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.870+0000 7f7038ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:19.859 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:19.870+0000 7f7038ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:20.191 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.200+0000 7f70425ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:20.191 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.200+0000 7f70425ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:20.191 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.200+0000 7f70425ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:28:20.191 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.200+0000 7f70425ed640 1 -- 172.21.15.67:0/304562783 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f703c13b890 con 0x7f703c1457f0 2024-03-29T11:28:20.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.202+0000 7f7038ff9640 1 -- 172.21.15.67:0/304562783 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f70340526a0 con 0x7f703c1457f0 2024-03-29T11:28:20.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.202+0000 7f7038ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:20.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.202+0000 7f7038ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:20.193 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:28:20.193 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:20.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.207+0000 7f70425ed640 10 librados: watch_flush enter 2024-03-29T11:28:20.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.207+0000 7f70425ed640 10 librados: watch_flush exit 2024-03-29T11:28:20.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.207+0000 7f70425ed640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-29T11:28:20.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.207+0000 7f70425ed640 10 client.4530.objecter successfully canceled tick 2024-03-29T11:28:20.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.207+0000 7f70425ed640 1 -- 172.21.15.67:0/304562783 >> v1:172.21.15.67:6816/404787197 conn(0x7f702806dc50 legacy=0x7f7028070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.207+0000 7f70425ed640 10 monclient: shutdown 2024-03-29T11:28:20.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.207+0000 7f70425ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:20.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.207+0000 7f70425ed640 1 -- 172.21.15.67:0/304562783 >> v1:172.21.15.67:6789/0 conn(0x7f703c1457f0 legacy=0x7f703c1da550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.207+0000 7f70425ed640 1 -- 172.21.15.67:0/304562783 shutdown_connections 2024-03-29T11:28:20.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.207+0000 7f70425ed640 1 -- 172.21.15.67:0/304562783 >> 172.21.15.67:0/304562783 conn(0x7f703c066a00 msgr2=0x7f703c0b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:20.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.207+0000 7f70425ed640 1 -- 172.21.15.67:0/304562783 shutdown_connections 2024-03-29T11:28:20.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.208+0000 7f70425ed640 1 -- 172.21.15.67:0/304562783 wait complete. 2024-03-29T11:28:20.199 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:20.208+0000 7f70425ed640 1 librados: shutdown 2024-03-29T11:28:20.216 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:27.023 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:27.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.186+0000 7f35ef693640 10 monclient: get_monmap_and_config 2024-03-29T11:28:27.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.186+0000 7f35ef693640 10 monclient: build_initial_monmap 2024-03-29T11:28:27.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.186+0000 7f35ef693640 10 monclient: monmap: 2024-03-29T11:28:27.176 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:27.176 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:27.176 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:28:27.187483+0000 2024-03-29T11:28:27.176 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:28:27.187483+0000 2024-03-29T11:28:27.176 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:27.177 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:27.177 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:27.177 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:28:27.177 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:28:27.177 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:27.177 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.187+0000 7f35ef693640 1 Processor -- start 2024-03-29T11:28:27.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.187+0000 7f35ef693640 1 -- start start 2024-03-29T11:28:27.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.187+0000 7f35ef693640 10 monclient: init 2024-03-29T11:28:27.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.188+0000 7f35ef693640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:27.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.188+0000 7f35ef693640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:27.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.188+0000 7f35ef693640 10 monclient(hunting): picked mon.noname-c con 0x7f35e81422b0 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:27.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.188+0000 7f35ef693640 10 monclient(hunting): picked mon.noname-b con 0x7f35e8142690 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:27.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.188+0000 7f35ef693640 10 monclient(hunting): picked mon.noname-a con 0x7f35e80b17d0 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:27.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.188+0000 7f35ef693640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e8142c80 con 0x7f35e80b17d0 2024-03-29T11:28:27.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.188+0000 7f35ef693640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e80aae40 con 0x7f35e81422b0 2024-03-29T11:28:27.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.188+0000 7f35ef693640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e813fbd0 con 0x7f35e8142690 2024-03-29T11:28:27.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.188+0000 7f35ef693640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:27.181 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.188+0000 7f35ef693640 10 monclient(hunting): authenticate will time out at 1367.484480s 2024-03-29T11:28:27.181 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.190+0000 7f35ed42f640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f35e81422b0 0x7f35e813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:33982/0 (socket says 172.21.15.67:33982) 2024-03-29T11:28:27.181 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.190+0000 7f35ed42f640 1 -- 172.21.15.67:0/162139956 learned_addr learned my addr 172.21.15.67:0/162139956 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:27.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.191+0000 7f35e7fff640 1 -- 172.21.15.67:0/162139956 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315080495 0 0) 0x7f35e80aae40 con 0x7f35e81422b0 2024-03-29T11:28:27.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.191+0000 7f35e7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.191+0000 7f35e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.191+0000 7f35e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.186 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.191+0000 7f35e7fff640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:28:27.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.191+0000 7f35e7fff640 1 -- 172.21.15.67:0/162139956 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d0003520 con 0x7f35e81422b0 2024-03-29T11:28:27.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.191+0000 7f35e7fff640 1 -- 172.21.15.67:0/162139956 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1845154013 0 0) 0x7f35d0003520 con 0x7f35e81422b0 2024-03-29T11:28:27.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.191+0000 7f35e7fff640 1 -- 172.21.15.67:0/162139956 >> v1:172.21.15.67:6790/0 conn(0x7f35e8142690 legacy=0x7f35e80b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:27.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35e7fff640 1 -- 172.21.15.67:0/162139956 >> v1:172.21.15.67:6789/0 conn(0x7f35e80b17d0 legacy=0x7f35e80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:27.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35e7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:27.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35e7fff640 1 monclient: found mon.noname-c 2024-03-29T11:28:27.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:27.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35e7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:28:27.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35e7fff640 1 -- 172.21.15.67:0/162139956 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e8151940 con 0x7f35e81422b0 2024-03-29T11:28:27.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35e7fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:27.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35e7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:27.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:27.187 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35e7fff640 1 -- 172.21.15.67:0/162139956 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f35e0001690 con 0x7f35e81422b0 2024-03-29T11:28:27.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:27.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35e7fff640 10 monclient: dump: 2024-03-29T11:28:27.188 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:27.188 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:27.188 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:27.188 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:27.188 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.188 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:27.188 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:27.188 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:27.188 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:27.188 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:27.188 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35ef693640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:28:27.189 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35ef693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:27.189 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.192+0000 7f35e7fff640 1 -- 172.21.15.67:0/162139956 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35e00017f0 con 0x7f35e81422b0 2024-03-29T11:28:27.189 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.193+0000 7f35e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:27.189 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.193+0000 7f35e7fff640 1 -- 172.21.15.67:0/162139956 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f35e0001c40 con 0x7f35e81422b0 2024-03-29T11:28:27.189 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.193+0000 7f35e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.189 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.193+0000 7f35e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:27.189 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.193+0000 7f35e7fff640 10 monclient: dump: 2024-03-29T11:28:27.190 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:27.190 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:27.190 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:27.190 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:27.190 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.190 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:27.190 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:27.190 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:27.190 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:27.190 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:27.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.193+0000 7f35ef693640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:27.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.193+0000 7f35ef693640 4 set_mon_vals no callback set 2024-03-29T11:28:27.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.194+0000 7f35ef693640 10 monclient: shutdown 2024-03-29T11:28:27.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.194+0000 7f35ef693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:27.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.194+0000 7f35ef693640 1 -- 172.21.15.67:0/162139956 >> v1:172.21.15.82:6789/0 conn(0x7f35e81422b0 legacy=0x7f35e813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.194+0000 7f35ef693640 1 -- 172.21.15.67:0/162139956 shutdown_connections 2024-03-29T11:28:27.192 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.194+0000 7f35ef693640 1 -- 172.21.15.67:0/162139956 >> 172.21.15.67:0/162139956 conn(0x7f35e8066a00 msgr2=0x7f35e8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:27.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.195+0000 7f35ef693640 1 -- 172.21.15.67:0/162139956 shutdown_connections 2024-03-29T11:28:27.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.195+0000 7f35ef693640 1 -- 172.21.15.67:0/162139956 wait complete. 2024-03-29T11:28:27.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.196+0000 7f35ef693640 10 monclient: build_initial_monmap 2024-03-29T11:28:27.193 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.196+0000 7f35ef693640 10 monclient: monmap: 2024-03-29T11:28:27.193 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:27.193 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:27.193 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:28:27.193 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:28:27.193 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:27.193 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:27.194 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:27.194 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:28:27.194 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:28:27.194 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:27.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.196+0000 7f35ef693640 1 librados: starting msgr at 2024-03-29T11:28:27.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.196+0000 7f35ef693640 1 librados: starting objecter 2024-03-29T11:28:27.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.196+0000 7f35ef693640 1 Processor -- start 2024-03-29T11:28:27.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.196+0000 7f35ef693640 1 -- start start 2024-03-29T11:28:27.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.196+0000 7f35ef693640 1 librados: setting wanted keys 2024-03-29T11:28:27.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.196+0000 7f35ef693640 1 librados: calling monclient init 2024-03-29T11:28:27.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.196+0000 7f35ef693640 10 monclient: init 2024-03-29T11:28:27.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.197+0000 7f35ef693640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:27.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.197+0000 7f35ef693640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:27.194 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.197+0000 7f35ef693640 10 monclient(hunting): picked mon.noname-c con 0x7f35e80b17d0 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:27.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.197+0000 7f35ef693640 10 monclient(hunting): picked mon.noname-a con 0x7f35e81422b0 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:27.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.197+0000 7f35ef693640 10 monclient(hunting): picked mon.noname-b con 0x7f35e8142690 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:27.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.197+0000 7f35ef693640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e80b7600 con 0x7f35e81422b0 2024-03-29T11:28:27.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.197+0000 7f35ef693640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e806e910 con 0x7f35e8142690 2024-03-29T11:28:27.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.197+0000 7f35ef693640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e806eb80 con 0x7f35e80b17d0 2024-03-29T11:28:27.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.197+0000 7f35ef693640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:27.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.197+0000 7f35ef693640 10 monclient(hunting): authenticate will time out at 1367.493715s 2024-03-29T11:28:27.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.198+0000 7f35edc30640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f35e8142690 0x7f35e813e0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:34842/0 (socket says 172.21.15.67:34842) 2024-03-29T11:28:27.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.198+0000 7f35edc30640 1 -- 172.21.15.67:0/2443869226 learned_addr learned my addr 172.21.15.67:0/2443869226 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:27.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.199+0000 7f35e5ffb640 10 client.?.objecter ms_handle_connect 0x7f35e8142690 2024-03-29T11:28:27.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.199+0000 7f35e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:27.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262584689 0 0) 0x7f35e806e910 con 0x7f35e8142690 2024-03-29T11:28:27.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.195 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:28:27.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d4004eb0 con 0x7f35e8142690 2024-03-29T11:28:27.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 10 client.?.objecter ms_handle_connect 0x7f35e80b17d0 2024-03-29T11:28:27.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:27.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083265025 0 0) 0x7f35e806eb80 con 0x7f35e80b17d0 2024-03-29T11:28:27.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.196 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:28:27.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e806e910 con 0x7f35e80b17d0 2024-03-29T11:28:27.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2200212756 0 0) 0x7f35d4004eb0 con 0x7f35e8142690 2024-03-29T11:28:27.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35e806eb80 con 0x7f35e8142690 2024-03-29T11:28:27.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f35d8004380 con 0x7f35e8142690 2024-03-29T11:28:27.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:27.197 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:27.197 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:27.197 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:27.197 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:27.198 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:27.198 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.198 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:27.198 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:27.198 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:27.198 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:27.198 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:27.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.200+0000 7f35e5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:27.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.201+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1785193619 0 0) 0x7f35e806eb80 con 0x7f35e8142690 2024-03-29T11:28:27.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.201+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 >> v1:172.21.15.67:6790/0 conn(0x7f35e80b17d0 legacy=0x7f35e80b6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.198 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.201+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 >> v1:172.21.15.67:6789/0 conn(0x7f35e81422b0 legacy=0x7f35e813d9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:27.200 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:27.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e5ffb640 1 monclient: found mon.b 2024-03-29T11:28:27.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:27.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:28:27.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e806ee60 con 0x7f35e8142690 2024-03-29T11:28:27.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:27.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:27.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:27.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35ef693640 5 monclient: authenticate success, global_id 4498 2024-03-29T11:28:27.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35ef693640 10 monclient: _renew_subs 2024-03-29T11:28:27.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35ef693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:28:27.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35ef693640 1 -- 172.21.15.67:0/2443869226 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e80b6250 con 0x7f35e8142690 2024-03-29T11:28:27.201 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35d8001250 con 0x7f35e8142690 2024-03-29T11:28:27.203 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:27.203 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f35d8001620 con 0x7f35e8142690 2024-03-29T11:28:27.203 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.203 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:27.203 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e5ffb640 10 monclient: dump: 2024-03-29T11:28:27.204 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:27.204 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:27.204 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:27.204 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:27.204 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.204 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:27.204 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:27.204 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:27.204 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:27.204 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:27.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35ef693640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:27.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35ef693640 10 monclient: _renew_subs 2024-03-29T11:28:27.204 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35ef693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:28:27.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35ef693640 1 -- 172.21.15.67:0/2443869226 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e8071580 con 0x7f35e8142690 2024-03-29T11:28:27.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35ef693640 1 librados: init done 2024-03-29T11:28:27.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.203+0000 7f35e77fe640 4 set_mon_vals no callback set 2024-03-29T11:28:27.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.205+0000 7f35ef693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:27.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.205+0000 7f35ef693640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:27.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.205+0000 7f35ef693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:28:27.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.205+0000 7f35ef693640 1 -- 172.21.15.67:0/2443869226 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e8075f10 con 0x7f35e8142690 2024-03-29T11:28:27.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.205+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f35d8001780 con 0x7f35e8142690 2024-03-29T11:28:27.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.206+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f35d80a6d90 con 0x7f35e8142690 2024-03-29T11:28:27.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.206+0000 7f35e5ffb640 10 client.4498.objecter ms_dispatch 0x7f35e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:27.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.206+0000 7f35e5ffb640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:27.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.206+0000 7f35e5ffb640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:27.205 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.206+0000 7f35e5ffb640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-29T11:28:27.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.206+0000 7f35ed42f640 10 monclient: get_auth_request con 0x7f35d406dd40 auth_method 0 2024-03-29T11:28:27.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.206+0000 7f35e5ffb640 10 client.4498.objecter ms_handle_connect 0x7f35d406dd40 2024-03-29T11:28:27.206 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.214+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35d8072670 con 0x7f35e8142690 2024-03-29T11:28:27.207 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.214+0000 7f35e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:27.207 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.214+0000 7f35e5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:27.521 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.532+0000 7f35ef693640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:27.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.532+0000 7f35ef693640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:27.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.532+0000 7f35ef693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:28:27.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.532+0000 7f35ef693640 1 -- 172.21.15.67:0/2443869226 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35e80b2270 con 0x7f35e8142690 2024-03-29T11:28:27.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.534+0000 7f35e5ffb640 1 -- 172.21.15.67:0/2443869226 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f35d80745a0 con 0x7f35e8142690 2024-03-29T11:28:27.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.534+0000 7f35e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:27.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.534+0000 7f35e5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:27.526 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:28:27.526 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:27.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.539+0000 7f35ef693640 10 librados: watch_flush enter 2024-03-29T11:28:27.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.539+0000 7f35ef693640 10 librados: watch_flush exit 2024-03-29T11:28:27.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.539+0000 7f35ef693640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-29T11:28:27.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.539+0000 7f35ef693640 10 client.4498.objecter successfully canceled tick 2024-03-29T11:28:27.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.539+0000 7f35ef693640 1 -- 172.21.15.67:0/2443869226 >> v1:172.21.15.67:6816/404787197 conn(0x7f35d406dd40 legacy=0x7f35d4070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.540+0000 7f35ef693640 10 monclient: shutdown 2024-03-29T11:28:27.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.540+0000 7f35ef693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:27.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.540+0000 7f35ef693640 1 -- 172.21.15.67:0/2443869226 >> v1:172.21.15.82:6789/0 conn(0x7f35e8142690 legacy=0x7f35e813e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.541+0000 7f35ef693640 1 -- 172.21.15.67:0/2443869226 shutdown_connections 2024-03-29T11:28:27.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.541+0000 7f35ef693640 1 -- 172.21.15.67:0/2443869226 >> 172.21.15.67:0/2443869226 conn(0x7f35e8066a00 msgr2=0x7f35e80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:27.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.541+0000 7f35ef693640 1 -- 172.21.15.67:0/2443869226 shutdown_connections 2024-03-29T11:28:27.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.541+0000 7f35ef693640 1 -- 172.21.15.67:0/2443869226 wait complete. 2024-03-29T11:28:27.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:27.541+0000 7f35ef693640 1 librados: shutdown 2024-03-29T11:28:27.548 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:34.356 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:34.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.517+0000 7f7434663640 10 monclient: get_monmap_and_config 2024-03-29T11:28:34.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.517+0000 7f7434663640 10 monclient: build_initial_monmap 2024-03-29T11:28:34.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.517+0000 7f7434663640 10 monclient: monmap: 2024-03-29T11:28:34.509 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:34.509 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:34.509 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:28:34.518424+0000 2024-03-29T11:28:34.509 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:28:34.518424+0000 2024-03-29T11:28:34.509 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:34.509 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:34.510 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:34.510 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:28:34.510 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:28:34.510 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:34.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.519+0000 7f7434663640 1 Processor -- start 2024-03-29T11:28:34.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.520+0000 7f7434663640 1 -- start start 2024-03-29T11:28:34.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.520+0000 7f7434663640 10 monclient: init 2024-03-29T11:28:34.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.521+0000 7f7434663640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:34.511 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.521+0000 7f7434663640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:34.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.521+0000 7f7434663640 10 monclient(hunting): picked mon.noname-a con 0x7f742c0a9ac0 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:34.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.522+0000 7f7434663640 10 monclient(hunting): picked mon.noname-b con 0x7f742c13ee90 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:34.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.522+0000 7f7434663640 10 monclient(hunting): picked mon.noname-c con 0x7f742c0af900 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:34.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.522+0000 7f7434663640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f742c0afd10 con 0x7f742c0a9ac0 2024-03-29T11:28:34.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.522+0000 7f7434663640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f742c0af6f0 con 0x7f742c0af900 2024-03-29T11:28:34.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.522+0000 7f7434663640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f742c0aac80 con 0x7f742c13ee90 2024-03-29T11:28:34.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.522+0000 7f7434663640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:34.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.522+0000 7f7434663640 10 monclient(hunting): authenticate will time out at 1374.818075s 2024-03-29T11:28:34.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.522+0000 7f7431bfe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f742c13ee90 0x7f742c13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:56580/0 (socket says 172.21.15.67:56580) 2024-03-29T11:28:34.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.522+0000 7f7431bfe640 1 -- 172.21.15.67:0/3952679348 learned_addr learned my addr 172.21.15.67:0/3952679348 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:34.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.523+0000 7f74313fd640 1 -- 172.21.15.67:0/3952679348 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965526928 0 0) 0x7f742c0aac80 con 0x7f742c13ee90 2024-03-29T11:28:34.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.523+0000 7f74313fd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:34.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.523+0000 7f74313fd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:34.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.523+0000 7f74313fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:34.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.523+0000 7f74313fd640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:28:34.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.523+0000 7f74313fd640 1 -- 172.21.15.67:0/3952679348 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7418003520 con 0x7f742c13ee90 2024-03-29T11:28:34.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.524+0000 7f74313fd640 1 -- 172.21.15.67:0/3952679348 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3131581404 0 0) 0x7f7418003520 con 0x7f742c13ee90 2024-03-29T11:28:34.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.524+0000 7f74313fd640 1 -- 172.21.15.67:0/3952679348 >> v1:172.21.15.82:6789/0 conn(0x7f742c0af900 legacy=0x7f742c149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:34.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.526+0000 7f74313fd640 1 -- 172.21.15.67:0/3952679348 >> v1:172.21.15.67:6789/0 conn(0x7f742c0a9ac0 legacy=0x7f742c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.526+0000 7f74313fd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:34.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.526+0000 7f74313fd640 1 monclient: found mon.noname-b 2024-03-29T11:28:34.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.526+0000 7f74313fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:34.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.526+0000 7f74313fd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:28:34.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.527+0000 7f74313fd640 1 -- 172.21.15.67:0/3952679348 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742c0aaf60 con 0x7f742c13ee90 2024-03-29T11:28:34.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.527+0000 7f74313fd640 10 monclient: _finish_auth 0 2024-03-29T11:28:34.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.527+0000 7f74313fd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:34.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.527+0000 7f74313fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:34.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.527+0000 7f74313fd640 1 -- 172.21.15.67:0/3952679348 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f741c001690 con 0x7f742c13ee90 2024-03-29T11:28:34.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.527+0000 7f7434663640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:28:34.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.527+0000 7f7434663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:34.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.527+0000 7f74313fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:34.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.527+0000 7f74313fd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:34.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.527+0000 7f74313fd640 10 monclient: dump: 2024-03-29T11:28:34.519 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:34.519 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:34.519 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:34.519 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:34.519 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:34.519 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:34.519 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:34.519 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:34.519 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:34.519 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:34.519 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.527+0000 7f7434663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:34.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.528+0000 7f74313fd640 1 -- 172.21.15.67:0/3952679348 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f741c001920 con 0x7f742c13ee90 2024-03-29T11:28:34.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.528+0000 7f74313fd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:34.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.528+0000 7f74313fd640 1 -- 172.21.15.67:0/3952679348 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f741c001cb0 con 0x7f742c13ee90 2024-03-29T11:28:34.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.528+0000 7f7434663640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:34.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.528+0000 7f7434663640 4 set_mon_vals no callback set 2024-03-29T11:28:34.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.529+0000 7f7434663640 10 monclient: shutdown 2024-03-29T11:28:34.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.529+0000 7f74313fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:34.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.530+0000 7f74313fd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:34.522 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.530+0000 7f74313fd640 10 monclient: dump: 2024-03-29T11:28:34.523 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:34.523 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:34.523 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:34.523 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:34.523 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:34.523 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:34.523 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:34.523 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:34.523 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:34.523 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:34.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.530+0000 7f7434663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:34.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.530+0000 7f7434663640 1 -- 172.21.15.67:0/3952679348 >> v1:172.21.15.67:6790/0 conn(0x7f742c13ee90 legacy=0x7f742c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.531+0000 7f7434663640 1 -- 172.21.15.67:0/3952679348 shutdown_connections 2024-03-29T11:28:34.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.531+0000 7f7434663640 1 -- 172.21.15.67:0/3952679348 >> 172.21.15.67:0/3952679348 conn(0x7f742c0b35e0 msgr2=0x7f742c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:34.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.531+0000 7f7434663640 1 -- 172.21.15.67:0/3952679348 shutdown_connections 2024-03-29T11:28:34.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.532+0000 7f7434663640 1 -- 172.21.15.67:0/3952679348 wait complete. 2024-03-29T11:28:34.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.533+0000 7f7434663640 10 monclient: build_initial_monmap 2024-03-29T11:28:34.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.533+0000 7f7434663640 10 monclient: monmap: 2024-03-29T11:28:34.527 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:34.527 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:34.527 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:28:34.527 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:28:34.527 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:34.527 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:34.527 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:34.527 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:28:34.527 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:28:34.527 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:34.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.533+0000 7f7434663640 1 librados: starting msgr at 2024-03-29T11:28:34.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.533+0000 7f7434663640 1 librados: starting objecter 2024-03-29T11:28:34.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.534+0000 7f7434663640 1 Processor -- start 2024-03-29T11:28:34.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.534+0000 7f7434663640 1 -- start start 2024-03-29T11:28:34.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.534+0000 7f7434663640 1 librados: setting wanted keys 2024-03-29T11:28:34.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.534+0000 7f7434663640 1 librados: calling monclient init 2024-03-29T11:28:34.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.534+0000 7f7434663640 10 monclient: init 2024-03-29T11:28:34.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.535+0000 7f7434663640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:34.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.535+0000 7f7434663640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:34.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.535+0000 7f7434663640 10 monclient(hunting): picked mon.noname-a con 0x7f742c0a9ac0 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:34.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.535+0000 7f7434663640 10 monclient(hunting): picked mon.noname-c con 0x7f742c0af900 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:34.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.535+0000 7f7434663640 10 monclient(hunting): picked mon.noname-b con 0x7f742c13ee90 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:34.528 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.535+0000 7f7434663640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f742c142320 con 0x7f742c0a9ac0 2024-03-29T11:28:34.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.535+0000 7f7434663640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f742c06ec20 con 0x7f742c13ee90 2024-03-29T11:28:34.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.535+0000 7f7434663640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f742c06e940 con 0x7f742c0af900 2024-03-29T11:28:34.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.535+0000 7f7434663640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:34.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7434663640 10 monclient(hunting): authenticate will time out at 1374.831912s 2024-03-29T11:28:34.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7431bfe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f742c0af900 0x7f742c14de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:58906/0 (socket says 172.21.15.67:58906) 2024-03-29T11:28:34.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7431bfe640 1 -- 172.21.15.67:0/2174064989 learned_addr learned my addr 172.21.15.67:0/2174064989 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:34.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 10 client.?.objecter ms_handle_connect 0x7f742c0af900 2024-03-29T11:28:34.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:34.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872908638 0 0) 0x7f742c06e940 con 0x7f742c0af900 2024-03-29T11:28:34.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:34.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:34.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:34.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:28:34.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74100070c0 con 0x7f742c0af900 2024-03-29T11:28:34.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 739905795 0 0) 0x7f74100070c0 con 0x7f742c0af900 2024-03-29T11:28:34.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f742c06e940 con 0x7f742c0af900 2024-03-29T11:28:34.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f741c001650 con 0x7f742c0af900 2024-03-29T11:28:34.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:34.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:34.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:34.532 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:34.532 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:34.532 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:34.532 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:34.532 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:34.532 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:34.533 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:34.533 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:34.533 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:34.533 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:34.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.536+0000 7f7422ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:34.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.537+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3257563495 0 0) 0x7f742c06e940 con 0x7f742c0af900 2024-03-29T11:28:34.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.537+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 >> v1:172.21.15.82:6789/0 conn(0x7f742c13ee90 legacy=0x7f742c06f040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:34.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 >> v1:172.21.15.67:6789/0 conn(0x7f742c0a9ac0 legacy=0x7f742c1411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:34.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 1 monclient: found mon.c 2024-03-29T11:28:34.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:34.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:34.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742c140480 con 0x7f742c0af900 2024-03-29T11:28:34.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:34.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:34.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:34.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7434663640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:28:34.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 10 client.?.objecter ms_handle_connect 0x7f742c0a9ac0 2024-03-29T11:28:34.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 10 client.4592.objecter resend_mon_ops 2024-03-29T11:28:34.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7434663640 10 monclient: _renew_subs 2024-03-29T11:28:34.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7434663640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:34.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7434663640 1 -- 172.21.15.67:0/2174064989 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742c130b40 con 0x7f742c0af900 2024-03-29T11:28:34.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7434663640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:34.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f741c004a90 con 0x7f742c0af900 2024-03-29T11:28:34.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f741c004e60 con 0x7f742c0af900 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7422ffd640 10 monclient: dump: 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:34.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7423fff640 4 set_mon_vals no callback set 2024-03-29T11:28:34.536 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7434663640 10 monclient: _renew_subs 2024-03-29T11:28:34.536 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7434663640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:34.536 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7434663640 1 -- 172.21.15.67:0/2174064989 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f742c14e730 con 0x7f742c0af900 2024-03-29T11:28:34.536 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.540+0000 7f7434663640 1 librados: init done 2024-03-29T11:28:34.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.542+0000 7f7434663640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:34.537 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.542+0000 7f7434663640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:34.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.542+0000 7f7434663640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:34.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.542+0000 7f7434663640 1 -- 172.21.15.67:0/2174064989 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742c075860 con 0x7f742c0af900 2024-03-29T11:28:34.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.543+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f741c01ae30 con 0x7f742c0af900 2024-03-29T11:28:34.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.544+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f741c084b90 con 0x7f742c0af900 2024-03-29T11:28:34.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.544+0000 7f74323ff640 10 monclient: get_auth_request con 0x7f7410076390 auth_method 0 2024-03-29T11:28:34.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.544+0000 7f7422ffd640 10 client.4592.objecter ms_dispatch 0x7f742c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:34.538 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.545+0000 7f7422ffd640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:34.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.545+0000 7f7422ffd640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:34.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.545+0000 7f7422ffd640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-29T11:28:34.539 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.545+0000 7f7422ffd640 10 client.4592.objecter ms_handle_connect 0x7f7410076390 2024-03-29T11:28:34.541 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.551+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f741c050470 con 0x7f742c0af900 2024-03-29T11:28:34.541 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.551+0000 7f7422ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:34.541 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.551+0000 7f7422ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:34.877 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.888+0000 7f7434663640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:34.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.888+0000 7f7434663640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:34.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.888+0000 7f7434663640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:34.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.888+0000 7f7434663640 1 -- 172.21.15.67:0/2174064989 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f742c0aa900 con 0x7f742c0af900 2024-03-29T11:28:34.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.888+0000 7f7422ffd640 1 -- 172.21.15.67:0/2174064989 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f741c0523a0 con 0x7f742c0af900 2024-03-29T11:28:34.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.888+0000 7f7422ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:34.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.888+0000 7f7422ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:34.879 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:28:34.879 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:34.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.892+0000 7f7434663640 10 librados: watch_flush enter 2024-03-29T11:28:34.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.892+0000 7f7434663640 10 librados: watch_flush exit 2024-03-29T11:28:34.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.892+0000 7f7434663640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-29T11:28:34.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.892+0000 7f7434663640 10 client.4592.objecter successfully canceled tick 2024-03-29T11:28:34.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.893+0000 7f7434663640 1 -- 172.21.15.67:0/2174064989 >> v1:172.21.15.67:6816/404787197 conn(0x7f7410076390 legacy=0x7f7410078850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.893+0000 7f7434663640 10 monclient: shutdown 2024-03-29T11:28:34.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.893+0000 7f7434663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:34.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.893+0000 7f7434663640 1 -- 172.21.15.67:0/2174064989 >> v1:172.21.15.67:6790/0 conn(0x7f742c0af900 legacy=0x7f742c14de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.893+0000 7f7434663640 1 -- 172.21.15.67:0/2174064989 shutdown_connections 2024-03-29T11:28:34.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.893+0000 7f7434663640 1 -- 172.21.15.67:0/2174064989 >> 172.21.15.67:0/2174064989 conn(0x7f742c0b35e0 msgr2=0x7f742c124450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:34.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.893+0000 7f7434663640 1 -- 172.21.15.67:0/2174064989 shutdown_connections 2024-03-29T11:28:34.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.893+0000 7f7434663640 1 -- 172.21.15.67:0/2174064989 wait complete. 2024-03-29T11:28:34.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:34.893+0000 7f7434663640 1 librados: shutdown 2024-03-29T11:28:34.901 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:41.708 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:41.859 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.868+0000 7f35e759e640 10 monclient: get_monmap_and_config 2024-03-29T11:28:41.859 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.868+0000 7f35e759e640 10 monclient: build_initial_monmap 2024-03-29T11:28:41.859 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.868+0000 7f35e759e640 10 monclient: monmap: 2024-03-29T11:28:41.859 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:41.859 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:41.859 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:28:41.869881+0000 2024-03-29T11:28:41.859 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:28:41.869881+0000 2024-03-29T11:28:41.859 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:41.860 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:41.860 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:41.860 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:28:41.860 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:28:41.860 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:41.860 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.869+0000 7f35e759e640 1 Processor -- start 2024-03-29T11:28:41.860 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.869+0000 7f35e759e640 1 -- start start 2024-03-29T11:28:41.861 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.869+0000 7f35e759e640 10 monclient: init 2024-03-29T11:28:41.861 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.870+0000 7f35e759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:41.863 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.870+0000 7f35e759e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:41.863 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.870+0000 7f35e759e640 10 monclient(hunting): picked mon.noname-b con 0x7f35e00389e0 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:41.863 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.870+0000 7f35e759e640 10 monclient(hunting): picked mon.noname-a con 0x7f35e0038dc0 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:41.863 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.870+0000 7f35e759e640 10 monclient(hunting): picked mon.noname-c con 0x7f35e0030660 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:41.863 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.870+0000 7f35e759e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00c54c0 con 0x7f35e0038dc0 2024-03-29T11:28:41.863 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.870+0000 7f35e759e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e0036430 con 0x7f35e0030660 2024-03-29T11:28:41.863 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.870+0000 7f35e759e640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e0039840 con 0x7f35e00389e0 2024-03-29T11:28:41.863 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.870+0000 7f35e759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:41.863 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.870+0000 7f35e759e640 10 monclient(hunting): authenticate will time out at 1382.167726s 2024-03-29T11:28:41.865 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.874+0000 7f35e659c640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f35e00389e0 0x7f35e0033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:37901/0 (socket says 172.21.15.67:37901) 2024-03-29T11:28:41.865 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.874+0000 7f35e659c640 1 -- 172.21.15.67:0/3391394934 learned_addr learned my addr 172.21.15.67:0/3391394934 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:41.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 1 -- 172.21.15.67:0/3391394934 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026958168 0 0) 0x7f35e0039840 con 0x7f35e00389e0 2024-03-29T11:28:41.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:41.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:41.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:41.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:28:41.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 1 -- 172.21.15.67:0/3391394934 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d0003520 con 0x7f35e00389e0 2024-03-29T11:28:41.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 1 -- 172.21.15.67:0/3391394934 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 735201707 0 0) 0x7f35d0003520 con 0x7f35e00389e0 2024-03-29T11:28:41.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 1 -- 172.21.15.67:0/3391394934 >> v1:172.21.15.82:6789/0 conn(0x7f35e0030660 legacy=0x7f35e00d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:41.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 1 -- 172.21.15.67:0/3391394934 >> v1:172.21.15.67:6789/0 conn(0x7f35e0038dc0 legacy=0x7f35e00cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:41.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 10 monclient: _finish_hunting 0 2024-03-29T11:28:41.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 1 monclient: found mon.noname-b 2024-03-29T11:28:41.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:41.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:28:41.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 1 -- 172.21.15.67:0/3391394934 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e0035690 con 0x7f35e00389e0 2024-03-29T11:28:41.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 10 monclient: _finish_auth 0 2024-03-29T11:28:41.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 10 monclient: _check_auth_tickets 2024-03-29T11:28:41.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:41.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 1 -- 172.21.15.67:0/3391394934 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f35e804f730 con 0x7f35e00389e0 2024-03-29T11:28:41.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:41.873 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:41.873 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 10 monclient: dump: 2024-03-29T11:28:41.873 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:41.873 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:41.873 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:41.873 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:41.873 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:41.873 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:41.873 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:41.873 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:41.873 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:41.873 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:41.873 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.875+0000 7f35e559a640 1 -- 172.21.15.67:0/3391394934 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35e80008d0 con 0x7f35e00389e0 2024-03-29T11:28:41.873 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.876+0000 7f35e559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:41.874 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.876+0000 7f35e559a640 1 -- 172.21.15.67:0/3391394934 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f35e80758b0 con 0x7f35e00389e0 2024-03-29T11:28:41.874 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.876+0000 7f35e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:41.874 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.876+0000 7f35e559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:41.874 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.876+0000 7f35e559a640 10 monclient: dump: 2024-03-29T11:28:41.874 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:41.874 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:41.874 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:41.874 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:41.874 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:41.874 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:41.874 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:41.874 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:41.874 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:41.874 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:41.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.876+0000 7f35e759e640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:28:41.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.876+0000 7f35e759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:41.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.876+0000 7f35e759e640 4 set_mon_vals no callback set 2024-03-29T11:28:41.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.877+0000 7f35e759e640 10 monclient: shutdown 2024-03-29T11:28:41.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.877+0000 7f35e759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:41.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.877+0000 7f35e759e640 1 -- 172.21.15.67:0/3391394934 >> v1:172.21.15.67:6790/0 conn(0x7f35e00389e0 legacy=0x7f35e0033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:41.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.877+0000 7f35e759e640 1 -- 172.21.15.67:0/3391394934 shutdown_connections 2024-03-29T11:28:41.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.877+0000 7f35e759e640 1 -- 172.21.15.67:0/3391394934 >> 172.21.15.67:0/3391394934 conn(0x7f35e0005c00 msgr2=0x7f35e0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:41.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.877+0000 7f35e759e640 1 -- 172.21.15.67:0/3391394934 shutdown_connections 2024-03-29T11:28:41.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.878+0000 7f35e759e640 1 -- 172.21.15.67:0/3391394934 wait complete. 2024-03-29T11:28:41.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.879+0000 7f35e759e640 10 monclient: build_initial_monmap 2024-03-29T11:28:41.875 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.879+0000 7f35e759e640 10 monclient: monmap: 2024-03-29T11:28:41.875 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:41.876 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:41.876 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:28:41.876 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:28:41.876 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:41.881 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:41.881 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:41.881 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:28:41.881 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:28:41.881 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:41.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.879+0000 7f35e759e640 1 librados: starting msgr at 2024-03-29T11:28:41.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.879+0000 7f35e759e640 1 librados: starting objecter 2024-03-29T11:28:41.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.879+0000 7f35e759e640 1 Processor -- start 2024-03-29T11:28:41.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.879+0000 7f35e759e640 1 -- start start 2024-03-29T11:28:41.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.879+0000 7f35e759e640 1 librados: setting wanted keys 2024-03-29T11:28:41.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.879+0000 7f35e759e640 1 librados: calling monclient init 2024-03-29T11:28:41.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.879+0000 7f35e759e640 10 monclient: init 2024-03-29T11:28:41.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.880+0000 7f35e759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:41.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.880+0000 7f35e759e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:41.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.883+0000 7f35e759e640 10 monclient(hunting): picked mon.noname-b con 0x7f35e0030660 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:41.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.883+0000 7f35e759e640 10 monclient(hunting): picked mon.noname-a con 0x7f35e0160fb0 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:41.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.887+0000 7f35e659c640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f35e0030660 0x7f35e002fa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:38809/0 (socket says 172.21.15.67:38809) 2024-03-29T11:28:41.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.887+0000 7f35e659c640 1 -- 172.21.15.67:0/991264751 learned_addr learned my addr 172.21.15.67:0/991264751 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:41.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.887+0000 7f35d6ffd640 10 client.?.objecter ms_handle_connect 0x7f35e0030660 2024-03-29T11:28:41.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.887+0000 7f35d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:41.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.890+0000 7f35e759e640 10 monclient(hunting): picked mon.noname-c con 0x7f35e00c5900 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:41.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.890+0000 7f35d6ffd640 10 client.?.objecter ms_handle_connect 0x7f35e00c5900 2024-03-29T11:28:41.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.890+0000 7f35d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:41.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.890+0000 7f35e759e640 1 -- 172.21.15.67:0/991264751 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00c7510 con 0x7f35e0160fb0 2024-03-29T11:28:41.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.890+0000 7f35e759e640 1 -- 172.21.15.67:0/991264751 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e0164650 con 0x7f35e0030660 2024-03-29T11:28:41.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.890+0000 7f35e759e640 1 -- 172.21.15.67:0/991264751 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00c0c00 con 0x7f35e00c5900 2024-03-29T11:28:41.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.890+0000 7f35e759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:41.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.890+0000 7f35e759e640 10 monclient(hunting): authenticate will time out at 1382.187760s 2024-03-29T11:28:41.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 10 client.?.objecter ms_handle_connect 0x7f35e0160fb0 2024-03-29T11:28:41.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:41.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647214386 0 0) 0x7f35e0164650 con 0x7f35e0030660 2024-03-29T11:28:41.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:41.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:41.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:41.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:28:41.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c8004eb0 con 0x7f35e0030660 2024-03-29T11:28:41.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4251753139 0 0) 0x7f35c8004eb0 con 0x7f35e0030660 2024-03-29T11:28:41.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35e0164650 con 0x7f35e0030660 2024-03-29T11:28:41.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f35e804f400 con 0x7f35e0030660 2024-03-29T11:28:41.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:41.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:41.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:41.889 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:41.889 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:41.889 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:41.889 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:41.889 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:41.889 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:41.889 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:41.889 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:41.889 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:41.889 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:41.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.891+0000 7f35d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:41.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.892+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601243340 0 0) 0x7f35e00c7510 con 0x7f35e0160fb0 2024-03-29T11:28:41.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.892+0000 7f35d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:41.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.892+0000 7f35d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:41.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.892+0000 7f35d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:41.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.892+0000 7f35d6ffd640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:28:41.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.892+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c8004eb0 con 0x7f35e0160fb0 2024-03-29T11:28:41.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.893+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2919557602 0 0) 0x7f35c8004eb0 con 0x7f35e0160fb0 2024-03-29T11:28:41.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.893+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35c80058a0 con 0x7f35e0160fb0 2024-03-29T11:28:41.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.893+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f35d80017d0 con 0x7f35e0160fb0 2024-03-29T11:28:41.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.893+0000 7f35d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:41.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.893+0000 7f35d6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:41.890 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.893+0000 7f35d6ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:41.890 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:41.890 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:41.891 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:41.891 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:41.891 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:41.891 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:41.891 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:41.891 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:41.891 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:41.891 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:41.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.896+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3779486479 0 0) 0x7f35c80058a0 con 0x7f35e0160fb0 2024-03-29T11:28:41.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.896+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 >> v1:172.21.15.67:6790/0 conn(0x7f35e00c5900 legacy=0x7f35e01616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:41.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.896+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 >> v1:172.21.15.82:6789/0 conn(0x7f35e0030660 legacy=0x7f35e002fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:41.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.896+0000 7f35d6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:41.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.896+0000 7f35d6ffd640 1 monclient: found mon.a 2024-03-29T11:28:41.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.896+0000 7f35d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:41.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.896+0000 7f35d6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:28:41.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.896+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e00c5650 con 0x7f35e0160fb0 2024-03-29T11:28:41.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.896+0000 7f35d6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:41.894 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.896+0000 7f35d6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:41.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.896+0000 7f35d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:41.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35e759e640 5 monclient: authenticate success, global_id 4539 2024-03-29T11:28:41.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35e759e640 10 monclient: _renew_subs 2024-03-29T11:28:41.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35e759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:28:41.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35e759e640 1 -- 172.21.15.67:0/991264751 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e0165280 con 0x7f35e0160fb0 2024-03-29T11:28:41.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35e759e640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:41.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35e759e640 10 monclient: _renew_subs 2024-03-29T11:28:41.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35e759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:28:41.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35e759e640 1 -- 172.21.15.67:0/991264751 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e016d800 con 0x7f35e0160fb0 2024-03-29T11:28:41.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35e759e640 1 librados: init done 2024-03-29T11:28:41.895 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35d8001c80 con 0x7f35e0160fb0 2024-03-29T11:28:41.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:41.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f35d80017d0 con 0x7f35e0160fb0 2024-03-29T11:28:41.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:41.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:41.896 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35d6ffd640 10 monclient: dump: 2024-03-29T11:28:41.896 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:41.896 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:41.896 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:41.896 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:41.896 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:41.896 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:41.896 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:41.896 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:41.896 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:41.897 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:41.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.897+0000 7f35d7fff640 4 set_mon_vals no callback set 2024-03-29T11:28:41.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.898+0000 7f35e759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:41.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.898+0000 7f35e759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:41.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.898+0000 7f35e759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:28:41.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.898+0000 7f35e759e640 1 -- 172.21.15.67:0/991264751 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35cc005080 con 0x7f35e0160fb0 2024-03-29T11:28:41.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.902+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f35d80035f0 con 0x7f35e0160fb0 2024-03-29T11:28:41.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.903+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f35d80832e0 con 0x7f35e0160fb0 2024-03-29T11:28:41.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.903+0000 7f35d6ffd640 10 client.4539.objecter ms_dispatch 0x7f35e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:41.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.903+0000 7f35d6ffd640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:41.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.903+0000 7f35d6ffd640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:41.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.903+0000 7f35d6ffd640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-29T11:28:41.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.903+0000 7f35e659c640 10 monclient: get_auth_request con 0x7f35c806e1d0 auth_method 0 2024-03-29T11:28:41.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.903+0000 7f35d6ffd640 10 client.4539.objecter ms_handle_connect 0x7f35c806e1d0 2024-03-29T11:28:41.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.910+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35d804ec20 con 0x7f35e0160fb0 2024-03-29T11:28:41.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.910+0000 7f35d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:41.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:41.910+0000 7f35d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:42.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.249+0000 7f35e759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:42.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.249+0000 7f35e759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:42.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.249+0000 7f35e759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:28:42.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.249+0000 7f35e759e640 1 -- 172.21.15.67:0/991264751 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35cc005330 con 0x7f35e0160fb0 2024-03-29T11:28:42.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.250+0000 7f35d6ffd640 1 -- 172.21.15.67:0/991264751 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f35d8082990 con 0x7f35e0160fb0 2024-03-29T11:28:42.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.250+0000 7f35d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:42.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.250+0000 7f35d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:42.242 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:28:42.242 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:42.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.254+0000 7f35e759e640 10 librados: watch_flush enter 2024-03-29T11:28:42.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.254+0000 7f35e759e640 10 librados: watch_flush exit 2024-03-29T11:28:42.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.254+0000 7f35e759e640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-29T11:28:42.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.254+0000 7f35e759e640 10 client.4539.objecter successfully canceled tick 2024-03-29T11:28:42.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.254+0000 7f35e759e640 1 -- 172.21.15.67:0/991264751 >> v1:172.21.15.67:6816/404787197 conn(0x7f35c806e1d0 legacy=0x7f35c8070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.254+0000 7f35e759e640 10 monclient: shutdown 2024-03-29T11:28:42.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.254+0000 7f35e759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:42.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.254+0000 7f35e759e640 1 -- 172.21.15.67:0/991264751 >> v1:172.21.15.67:6789/0 conn(0x7f35e0160fb0 legacy=0x7f35e0164b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.254+0000 7f35e759e640 1 -- 172.21.15.67:0/991264751 shutdown_connections 2024-03-29T11:28:42.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.254+0000 7f35e759e640 1 -- 172.21.15.67:0/991264751 >> 172.21.15.67:0/991264751 conn(0x7f35e0005c00 msgr2=0x7f35e0034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:42.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.254+0000 7f35e759e640 1 -- 172.21.15.67:0/991264751 shutdown_connections 2024-03-29T11:28:42.246 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.254+0000 7f35e759e640 1 -- 172.21.15.67:0/991264751 wait complete. 2024-03-29T11:28:42.247 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:42.254+0000 7f35e759e640 1 librados: shutdown 2024-03-29T11:28:42.265 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:49.073 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:49.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.231+0000 7f75157a5640 10 monclient: get_monmap_and_config 2024-03-29T11:28:49.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.231+0000 7f75157a5640 10 monclient: build_initial_monmap 2024-03-29T11:28:49.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.232+0000 7f75157a5640 10 monclient: monmap: 2024-03-29T11:28:49.224 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:49.224 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:49.224 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:28:49.233777+0000 2024-03-29T11:28:49.224 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:28:49.233777+0000 2024-03-29T11:28:49.224 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:49.224 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:49.224 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:49.225 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:28:49.225 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:28:49.225 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:49.226 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.232+0000 7f75157a5640 1 Processor -- start 2024-03-29T11:28:49.226 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.232+0000 7f75157a5640 1 -- start start 2024-03-29T11:28:49.226 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.232+0000 7f75157a5640 10 monclient: init 2024-03-29T11:28:49.226 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.233+0000 7f75157a5640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:49.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.233+0000 7f75157a5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:49.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.233+0000 7f75157a5640 10 monclient(hunting): picked mon.noname-b con 0x7f75101457f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:49.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.233+0000 7f75157a5640 10 monclient(hunting): picked mon.noname-c con 0x7f751013f650 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:49.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.233+0000 7f75157a5640 10 monclient(hunting): picked mon.noname-a con 0x7f75100a9d70 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:49.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.233+0000 7f75157a5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75100aaef0 con 0x7f75100a9d70 2024-03-29T11:28:49.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.233+0000 7f75157a5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7510145bd0 con 0x7f751013f650 2024-03-29T11:28:49.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.233+0000 7f75157a5640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7510145eb0 con 0x7f75101457f0 2024-03-29T11:28:49.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.233+0000 7f75157a5640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:49.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.233+0000 7f75157a5640 10 monclient(hunting): authenticate will time out at 1389.530809s 2024-03-29T11:28:49.231 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.238+0000 7f750f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f75100a9d70 0x7f751014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39955/0 (socket says 172.21.15.67:39955) 2024-03-29T11:28:49.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.238+0000 7f750f7fe640 1 -- 172.21.15.67:0/2523858955 learned_addr learned my addr 172.21.15.67:0/2523858955 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:49.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 1 -- 172.21.15.67:0/2523858955 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777150275 0 0) 0x7f75100aaef0 con 0x7f75100a9d70 2024-03-29T11:28:49.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:49.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:49.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:49.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:28:49.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 1 -- 172.21.15.67:0/2523858955 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f8003520 con 0x7f75100a9d70 2024-03-29T11:28:49.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 1 -- 172.21.15.67:0/2523858955 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1013605546 0 0) 0x7f74f8003520 con 0x7f75100a9d70 2024-03-29T11:28:49.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 1 -- 172.21.15.67:0/2523858955 >> v1:172.21.15.67:6790/0 conn(0x7f75101457f0 legacy=0x7f751013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:49.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 1 -- 172.21.15.67:0/2523858955 >> v1:172.21.15.82:6789/0 conn(0x7f751013f650 legacy=0x7f75100b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:49.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:49.232 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 1 monclient: found mon.noname-a 2024-03-29T11:28:49.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:49.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:28:49.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 1 -- 172.21.15.67:0/2523858955 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7510146340 con 0x7f75100a9d70 2024-03-29T11:28:49.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:49.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:49.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:49.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 1 -- 172.21.15.67:0/2523858955 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7508001a10 con 0x7f75100a9d70 2024-03-29T11:28:49.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.239+0000 7f750dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.240+0000 7f750dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:49.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.240+0000 7f750dffb640 10 monclient: dump: 2024-03-29T11:28:49.233 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:49.233 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:49.233 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:49.234 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:49.234 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.234 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:49.234 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:49.234 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:49.234 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:49.234 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:49.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.240+0000 7f750dffb640 1 -- 172.21.15.67:0/2523858955 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7508001d20 con 0x7f75100a9d70 2024-03-29T11:28:49.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.240+0000 7f750dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:49.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.240+0000 7f750dffb640 1 -- 172.21.15.67:0/2523858955 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f75080020b0 con 0x7f75100a9d70 2024-03-29T11:28:49.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.240+0000 7f750dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.240+0000 7f750dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:49.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.240+0000 7f750dffb640 10 monclient: dump: 2024-03-29T11:28:49.235 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:49.235 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:49.235 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:49.235 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:49.235 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.235 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:49.235 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:49.235 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:49.235 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:49.235 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:49.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.240+0000 7f75157a5640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:28:49.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.240+0000 7f75157a5640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:49.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.240+0000 7f75157a5640 4 set_mon_vals no callback set 2024-03-29T11:28:49.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.241+0000 7f75157a5640 10 monclient: shutdown 2024-03-29T11:28:49.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.241+0000 7f75157a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:49.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.241+0000 7f75157a5640 1 -- 172.21.15.67:0/2523858955 >> v1:172.21.15.67:6789/0 conn(0x7f75100a9d70 legacy=0x7f751014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.245+0000 7f75157a5640 1 -- 172.21.15.67:0/2523858955 shutdown_connections 2024-03-29T11:28:49.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.245+0000 7f75157a5640 1 -- 172.21.15.67:0/2523858955 >> 172.21.15.67:0/2523858955 conn(0x7f7510066a00 msgr2=0x7f7510066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:49.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.245+0000 7f75157a5640 1 -- 172.21.15.67:0/2523858955 shutdown_connections 2024-03-29T11:28:49.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.246+0000 7f75157a5640 1 -- 172.21.15.67:0/2523858955 wait complete. 2024-03-29T11:28:49.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.247+0000 7f75157a5640 10 monclient: build_initial_monmap 2024-03-29T11:28:49.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.247+0000 7f75157a5640 10 monclient: monmap: 2024-03-29T11:28:49.242 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:49.242 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:49.242 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:28:49.242 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:28:49.242 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:49.242 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:49.242 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:49.242 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:28:49.242 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:28:49.243 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:49.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.247+0000 7f75157a5640 1 librados: starting msgr at 2024-03-29T11:28:49.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.247+0000 7f75157a5640 1 librados: starting objecter 2024-03-29T11:28:49.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.247+0000 7f75157a5640 1 Processor -- start 2024-03-29T11:28:49.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.247+0000 7f75157a5640 1 -- start start 2024-03-29T11:28:49.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.247+0000 7f75157a5640 1 librados: setting wanted keys 2024-03-29T11:28:49.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.247+0000 7f75157a5640 1 librados: calling monclient init 2024-03-29T11:28:49.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.247+0000 7f75157a5640 10 monclient: init 2024-03-29T11:28:49.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.248+0000 7f75157a5640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:49.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.248+0000 7f75157a5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:49.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.248+0000 7f75157a5640 10 monclient(hunting): picked mon.noname-a con 0x7f75100a9d70 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:49.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.248+0000 7f75157a5640 10 monclient(hunting): picked mon.noname-b con 0x7f751013f650 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:49.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.248+0000 7f75157a5640 10 monclient(hunting): picked mon.noname-c con 0x7f75101457f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:49.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.248+0000 7f75157a5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75101452c0 con 0x7f75100a9d70 2024-03-29T11:28:49.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.248+0000 7f75157a5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7510135770 con 0x7f751013f650 2024-03-29T11:28:49.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.248+0000 7f75157a5640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7510140470 con 0x7f75101457f0 2024-03-29T11:28:49.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.248+0000 7f75157a5640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:49.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.248+0000 7f75157a5640 10 monclient(hunting): authenticate will time out at 1389.545548s 2024-03-29T11:28:49.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.254+0000 7f750f7fe640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f75101457f0 0x7f7510135060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:57787/0 (socket says 172.21.15.67:57787) 2024-03-29T11:28:49.245 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.254+0000 7f750f7fe640 1 -- 172.21.15.67:0/1145714826 learned_addr learned my addr 172.21.15.67:0/1145714826 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:49.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.255+0000 7f74f77fe640 10 client.?.objecter ms_handle_connect 0x7f75101457f0 2024-03-29T11:28:49.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.255+0000 7f74f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:49.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.255+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130215995 0 0) 0x7f7510140470 con 0x7f75101457f0 2024-03-29T11:28:49.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.255+0000 7f74f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:49.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.255+0000 7f74f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:49.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.256+0000 7f74f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:49.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.256+0000 7f74f77fe640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:28:49.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.256+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74fc004eb0 con 0x7f75101457f0 2024-03-29T11:28:49.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.256+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4036953415 0 0) 0x7f74fc004eb0 con 0x7f75101457f0 2024-03-29T11:28:49.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.256+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7510140470 con 0x7f75101457f0 2024-03-29T11:28:49.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.256+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f75080019d0 con 0x7f75101457f0 2024-03-29T11:28:49.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.256+0000 7f74f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.256+0000 7f74f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:49.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.256+0000 7f74f77fe640 10 monclient(hunting): dump: 2024-03-29T11:28:49.249 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:49.249 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:49.249 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:49.250 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:49.250 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.250 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:49.250 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:49.250 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:49.250 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:49.250 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:49.250 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.256+0000 7f74f77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:49.250 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.257+0000 7f74f77fe640 10 client.?.objecter ms_handle_connect 0x7f751013f650 2024-03-29T11:28:49.250 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.257+0000 7f74f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:49.251 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.257+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2050614261 0 0) 0x7f7510140470 con 0x7f75101457f0 2024-03-29T11:28:49.251 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.257+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 >> v1:172.21.15.82:6789/0 conn(0x7f751013f650 legacy=0x7f7510134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.251 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.257+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 >> v1:172.21.15.67:6789/0 conn(0x7f75100a9d70 legacy=0x7f75101da480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:49.251 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.258+0000 7f74f77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:49.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f74f77fe640 1 monclient: found mon.c 2024-03-29T11:28:49.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f74f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:49.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f74f77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:49.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7510144e00 con 0x7f75101457f0 2024-03-29T11:28:49.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f74f77fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:49.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f74f77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:49.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f74f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:49.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f75157a5640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:28:49.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f75157a5640 10 monclient: _renew_subs 2024-03-29T11:28:49.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f75157a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:49.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f75157a5640 1 -- 172.21.15.67:0/1145714826 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7510145050 con 0x7f75101457f0 2024-03-29T11:28:49.252 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f75157a5640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:49.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f75157a5640 10 monclient: _renew_subs 2024-03-29T11:28:49.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f75157a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:49.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f75157a5640 1 -- 172.21.15.67:0/1145714826 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f751013a040 con 0x7f75101457f0 2024-03-29T11:28:49.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.259+0000 7f75157a5640 1 librados: init done 2024-03-29T11:28:49.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.261+0000 7f75157a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:49.253 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.261+0000 7f75157a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:49.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.262+0000 7f75157a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:49.254 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.262+0000 7f75157a5640 1 -- 172.21.15.67:0/1145714826 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7510140600 con 0x7f75101457f0 2024-03-29T11:28:49.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.265+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7508001b30 con 0x7f75101457f0 2024-03-29T11:28:49.256 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.265+0000 7f74f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:49.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.265+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f7508005190 con 0x7f75101457f0 2024-03-29T11:28:49.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.265+0000 7f74f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.265+0000 7f74f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:49.257 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.265+0000 7f74f77fe640 10 monclient: dump: 2024-03-29T11:28:49.257 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:49.257 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:49.257 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:49.257 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:49.257 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.257 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:49.258 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:49.258 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:49.258 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:49.258 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:49.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.265+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f750801b1a0 con 0x7f75101457f0 2024-03-29T11:28:49.258 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.265+0000 7f750d7fa640 4 set_mon_vals no callback set 2024-03-29T11:28:49.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.267+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f7508085090 con 0x7f75101457f0 2024-03-29T11:28:49.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.267+0000 7f74f77fe640 10 client.4601.objecter ms_dispatch 0x7f7510000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:49.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.267+0000 7f74f77fe640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:49.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.267+0000 7f74f77fe640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:49.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.267+0000 7f74f77fe640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-29T11:28:49.261 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.267+0000 7f750effd640 10 monclient: get_auth_request con 0x7f74fc06dcc0 auth_method 0 2024-03-29T11:28:49.262 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.270+0000 7f74f77fe640 10 client.4601.objecter ms_handle_connect 0x7f74fc06dcc0 2024-03-29T11:28:49.263 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.271+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75080509f0 con 0x7f75101457f0 2024-03-29T11:28:49.263 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.271+0000 7f74f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:49.263 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.271+0000 7f74f77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:49.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.598+0000 7f75157a5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:49.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.598+0000 7f75157a5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:49.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.598+0000 7f75157a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:28:49.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.598+0000 7f75157a5640 1 -- 172.21.15.67:0/1145714826 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7510075f10 con 0x7f75101457f0 2024-03-29T11:28:49.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.599+0000 7f74f77fe640 1 -- 172.21.15.67:0/1145714826 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f7508052920 con 0x7f75101457f0 2024-03-29T11:28:49.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.599+0000 7f74f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:49.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.599+0000 7f74f77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:49.591 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:28:49.591 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:49.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.604+0000 7f75157a5640 10 librados: watch_flush enter 2024-03-29T11:28:49.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.604+0000 7f75157a5640 10 librados: watch_flush exit 2024-03-29T11:28:49.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.604+0000 7f75157a5640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-29T11:28:49.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.604+0000 7f75157a5640 10 client.4601.objecter successfully canceled tick 2024-03-29T11:28:49.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.604+0000 7f75157a5640 1 -- 172.21.15.67:0/1145714826 >> v1:172.21.15.67:6816/404787197 conn(0x7f74fc06dcc0 legacy=0x7f74fc070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.604+0000 7f75157a5640 10 monclient: shutdown 2024-03-29T11:28:49.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.604+0000 7f75157a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:49.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.604+0000 7f75157a5640 1 -- 172.21.15.67:0/1145714826 >> v1:172.21.15.67:6790/0 conn(0x7f75101457f0 legacy=0x7f7510135060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.604+0000 7f75157a5640 1 -- 172.21.15.67:0/1145714826 shutdown_connections 2024-03-29T11:28:49.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.604+0000 7f75157a5640 1 -- 172.21.15.67:0/1145714826 >> 172.21.15.67:0/1145714826 conn(0x7f7510066a00 msgr2=0x7f751013b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:49.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.604+0000 7f75157a5640 1 -- 172.21.15.67:0/1145714826 shutdown_connections 2024-03-29T11:28:49.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.604+0000 7f75157a5640 1 -- 172.21.15.67:0/1145714826 wait complete. 2024-03-29T11:28:49.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:49.605+0000 7f75157a5640 1 librados: shutdown 2024-03-29T11:28:49.618 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:56.425 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:56.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.587+0000 7f2607d96640 10 monclient: get_monmap_and_config 2024-03-29T11:28:56.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.587+0000 7f2607d96640 10 monclient: build_initial_monmap 2024-03-29T11:28:56.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.587+0000 7f2607d96640 10 monclient: monmap: 2024-03-29T11:28:56.578 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:56.578 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:56.579 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:28:56.588592+0000 2024-03-29T11:28:56.579 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:28:56.588592+0000 2024-03-29T11:28:56.579 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:56.579 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:56.579 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:56.579 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:28:56.579 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:28:56.579 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:56.581 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.588+0000 7f2607d96640 1 Processor -- start 2024-03-29T11:28:56.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.591+0000 7f2607d96640 1 -- start start 2024-03-29T11:28:56.582 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.591+0000 7f2607d96640 10 monclient: init 2024-03-29T11:28:56.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.593+0000 7f2607d96640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:56.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.593+0000 7f2607d96640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:56.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.593+0000 7f2607d96640 10 monclient(hunting): picked mon.noname-a con 0x7f26001457f0 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:56.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.593+0000 7f2607d96640 10 monclient(hunting): picked mon.noname-b con 0x7f260013f650 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:56.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.593+0000 7f2607d96640 10 monclient(hunting): picked mon.noname-c con 0x7f26000a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:56.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.593+0000 7f2607d96640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26000aaef0 con 0x7f26001457f0 2024-03-29T11:28:56.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.593+0000 7f2607d96640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2600145bd0 con 0x7f26000a9d70 2024-03-29T11:28:56.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.593+0000 7f2607d96640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2600145eb0 con 0x7f260013f650 2024-03-29T11:28:56.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.593+0000 7f2607d96640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:56.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.593+0000 7f2607d96640 10 monclient(hunting): authenticate will time out at 1396.889601s 2024-03-29T11:28:56.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.596+0000 7f2605b32640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f26001457f0 0x7f260013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41783/0 (socket says 172.21.15.67:41783) 2024-03-29T11:28:56.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.596+0000 7f2605b32640 1 -- 172.21.15.67:0/1213787678 learned_addr learned my addr 172.21.15.67:0/1213787678 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:56.593 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.600+0000 7f2604b30640 1 -- 172.21.15.67:0/1213787678 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123376990 0 0) 0x7f26000aaef0 con 0x7f26001457f0 2024-03-29T11:28:56.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.600+0000 7f2604b30640 10 monclient(hunting): _negotiate 2024-03-29T11:28:56.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.600+0000 7f2604b30640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:56.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.600+0000 7f2604b30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:56.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.600+0000 7f2604b30640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:28:56.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.600+0000 7f2604b30640 1 -- 172.21.15.67:0/1213787678 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ec003520 con 0x7f26001457f0 2024-03-29T11:28:56.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 1 -- 172.21.15.67:0/1213787678 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166716474 0 0) 0x7f2600145eb0 con 0x7f260013f650 2024-03-29T11:28:56.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient(hunting): _negotiate 2024-03-29T11:28:56.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:56.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:56.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:28:56.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 1 -- 172.21.15.67:0/1213787678 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26000aaef0 con 0x7f260013f650 2024-03-29T11:28:56.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 1 -- 172.21.15.67:0/1213787678 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315848625 0 0) 0x7f2600145bd0 con 0x7f26000a9d70 2024-03-29T11:28:56.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient(hunting): _negotiate 2024-03-29T11:28:56.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:56.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:56.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:28:56.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 1 -- 172.21.15.67:0/1213787678 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2600145eb0 con 0x7f26000a9d70 2024-03-29T11:28:56.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 1 -- 172.21.15.67:0/1213787678 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 381039922 0 0) 0x7f25ec003520 con 0x7f26001457f0 2024-03-29T11:28:56.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 1 -- 172.21.15.67:0/1213787678 >> v1:172.21.15.67:6790/0 conn(0x7f260013f650 legacy=0x7f26000b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 1 -- 172.21.15.67:0/1213787678 >> v1:172.21.15.82:6789/0 conn(0x7f26000a9d70 legacy=0x7f260014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient: _finish_hunting 0 2024-03-29T11:28:56.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 1 monclient: found mon.noname-a 2024-03-29T11:28:56.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:56.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:28:56.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 1 -- 172.21.15.67:0/1213787678 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2600146340 con 0x7f26001457f0 2024-03-29T11:28:56.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient: _finish_auth 0 2024-03-29T11:28:56.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient: _check_auth_tickets 2024-03-29T11:28:56.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 1 -- 172.21.15.67:0/1213787678 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f25f8001a10 con 0x7f26001457f0 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2604b30640 10 monclient: dump: 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:56.596 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2607d96640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:28:56.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.601+0000 7f2607d96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:56.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.602+0000 7f2604b30640 1 -- 172.21.15.67:0/1213787678 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25f8001b70 con 0x7f26001457f0 2024-03-29T11:28:56.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.602+0000 7f2604b30640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:56.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.602+0000 7f2607d96640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:56.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.602+0000 7f2607d96640 4 set_mon_vals no callback set 2024-03-29T11:28:56.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.602+0000 7f2604b30640 1 -- 172.21.15.67:0/1213787678 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f25f8001f80 con 0x7f26001457f0 2024-03-29T11:28:56.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.602+0000 7f2604b30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:56.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.602+0000 7f2604b30640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:56.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.602+0000 7f2604b30640 10 monclient: dump: 2024-03-29T11:28:56.597 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:56.597 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:56.598 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:56.598 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:56.598 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:56.598 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:56.598 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:56.598 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:56.598 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:56.598 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:56.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.602+0000 7f2607d96640 10 monclient: shutdown 2024-03-29T11:28:56.598 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.602+0000 7f2607d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:56.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.602+0000 7f2607d96640 1 -- 172.21.15.67:0/1213787678 >> v1:172.21.15.67:6789/0 conn(0x7f26001457f0 legacy=0x7f260013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.603+0000 7f2607d96640 1 -- 172.21.15.67:0/1213787678 shutdown_connections 2024-03-29T11:28:56.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.603+0000 7f2607d96640 1 -- 172.21.15.67:0/1213787678 >> 172.21.15.67:0/1213787678 conn(0x7f2600066a00 msgr2=0x7f2600066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:56.599 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.605+0000 7f2607d96640 1 -- 172.21.15.67:0/1213787678 shutdown_connections 2024-03-29T11:28:56.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.605+0000 7f2607d96640 1 -- 172.21.15.67:0/1213787678 wait complete. 2024-03-29T11:28:56.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.606+0000 7f2607d96640 10 monclient: build_initial_monmap 2024-03-29T11:28:56.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.606+0000 7f2607d96640 10 monclient: monmap: 2024-03-29T11:28:56.600 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:28:56.600 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:56.600 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:28:56.600 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:28:56.600 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:56.600 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:56.600 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:28:56.600 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:28:56.600 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:28:56.600 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:56.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.606+0000 7f2607d96640 1 librados: starting msgr at 2024-03-29T11:28:56.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.606+0000 7f2607d96640 1 librados: starting objecter 2024-03-29T11:28:56.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.606+0000 7f2607d96640 1 Processor -- start 2024-03-29T11:28:56.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.606+0000 7f2607d96640 1 -- start start 2024-03-29T11:28:56.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.606+0000 7f2607d96640 1 librados: setting wanted keys 2024-03-29T11:28:56.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.606+0000 7f2607d96640 1 librados: calling monclient init 2024-03-29T11:28:56.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.606+0000 7f2607d96640 10 monclient: init 2024-03-29T11:28:56.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.607+0000 7f2607d96640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:56.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.607+0000 7f2607d96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:56.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.607+0000 7f2607d96640 10 monclient(hunting): picked mon.noname-c con 0x7f26000a9d70 addr v1:172.21.15.67:6790/0 2024-03-29T11:28:56.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.607+0000 7f2607d96640 10 monclient(hunting): picked mon.noname-a con 0x7f260013f650 addr v1:172.21.15.67:6789/0 2024-03-29T11:28:56.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.607+0000 7f2607d96640 10 monclient(hunting): picked mon.noname-b con 0x7f26001457f0 addr v1:172.21.15.82:6789/0 2024-03-29T11:28:56.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.607+0000 7f2607d96640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26001cecb0 con 0x7f260013f650 2024-03-29T11:28:56.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.607+0000 7f2607d96640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26001daf40 con 0x7f26001457f0 2024-03-29T11:28:56.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.607+0000 7f2607d96640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26001397b0 con 0x7f26000a9d70 2024-03-29T11:28:56.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.607+0000 7f2607d96640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:56.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.607+0000 7f2607d96640 10 monclient(hunting): authenticate will time out at 1396.903587s 2024-03-29T11:28:56.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.610+0000 7f2605b32640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f26000a9d70 0x7f26001db3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:51050/0 (socket says 172.21.15.67:51050) 2024-03-29T11:28:56.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.610+0000 7f2605b32640 1 -- 172.21.15.67:0/2673048656 learned_addr learned my addr 172.21.15.67:0/2673048656 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:28:56.603 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.611+0000 7f25f67fc640 10 client.?.objecter ms_handle_connect 0x7f26000a9d70 2024-03-29T11:28:56.603 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.611+0000 7f25f67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:56.603 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.613+0000 7f25f67fc640 10 client.?.objecter ms_handle_connect 0x7f26001457f0 2024-03-29T11:28:56.603 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.613+0000 7f25f67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:56.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.613+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328981821 0 0) 0x7f26001daf40 con 0x7f26001457f0 2024-03-29T11:28:56.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.614+0000 7f25f67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:56.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.614+0000 7f25f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:56.605 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.614+0000 7f25f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:56.605 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.614+0000 7f25f67fc640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:28:56.605 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.614+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e4004eb0 con 0x7f26001457f0 2024-03-29T11:28:56.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.615+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1279114224 0 0) 0x7f25e4004eb0 con 0x7f26001457f0 2024-03-29T11:28:56.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.615+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26001daf40 con 0x7f26001457f0 2024-03-29T11:28:56.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.615+0000 7f25f67fc640 10 client.?.objecter ms_handle_connect 0x7f260013f650 2024-03-29T11:28:56.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.615+0000 7f25f67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:56.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.615+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f25f00018f0 con 0x7f26001457f0 2024-03-29T11:28:56.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.615+0000 7f25f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:56.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.615+0000 7f25f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:56.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.615+0000 7f25f67fc640 10 monclient(hunting): dump: 2024-03-29T11:28:56.609 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:56.609 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:56.609 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:56.609 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:56.609 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:56.609 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:56.609 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:56.609 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:56.609 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:56.609 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:56.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.615+0000 7f25f67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:56.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.615+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290963107 0 0) 0x7f26001397b0 con 0x7f26000a9d70 2024-03-29T11:28:56.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.615+0000 7f25f67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:56.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.615+0000 7f25f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:56.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.615+0000 7f25f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:56.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:28:56.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e4004eb0 con 0x7f26000a9d70 2024-03-29T11:28:56.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106148666 0 0) 0x7f26001cecb0 con 0x7f260013f650 2024-03-29T11:28:56.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:56.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:56.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:56.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:28:56.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26001397b0 con 0x7f260013f650 2024-03-29T11:28:56.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 414388524 0 0) 0x7f26001daf40 con 0x7f26001457f0 2024-03-29T11:28:56.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 >> v1:172.21.15.67:6790/0 conn(0x7f26000a9d70 legacy=0x7f26001db3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 >> v1:172.21.15.67:6789/0 conn(0x7f260013f650 legacy=0x7f2600138990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 1 monclient: found mon.b 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26001cef00 con 0x7f26001457f0 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f2607d96640 5 monclient: authenticate success, global_id 4510 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f2607d96640 10 monclient: _renew_subs 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f2607d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f2607d96640 1 -- 172.21.15.67:0/2673048656 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26001dab10 con 0x7f26001457f0 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f2607d96640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:56.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f2607d96640 10 monclient: _renew_subs 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f2607d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f2607d96640 1 -- 172.21.15.67:0/2673048656 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2600139d10 con 0x7f26001457f0 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f2607d96640 1 librados: init done 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25f0001ac0 con 0x7f26001457f0 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f25f00038c0 con 0x7f26001457f0 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.616+0000 7f25f67fc640 10 monclient: dump: 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:28:56.612 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:28:56.613 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:28:56.613 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:28:56.613 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:28:56.613 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.617+0000 7f25f7fff640 4 set_mon_vals no callback set 2024-03-29T11:28:56.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.618+0000 7f2607d96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:56.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.619+0000 7f2607d96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:56.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.619+0000 7f2607d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:28:56.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.619+0000 7f2607d96640 1 -- 172.21.15.67:0/2673048656 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e8005080 con 0x7f26001457f0 2024-03-29T11:28:56.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.622+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f25f0003a20 con 0x7f26001457f0 2024-03-29T11:28:56.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.623+0000 7f2605b32640 10 monclient: get_auth_request con 0x7f25e406de50 auth_method 0 2024-03-29T11:28:56.615 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.623+0000 7f25f67fc640 10 client.4510.objecter ms_handle_connect 0x7f25e406de50 2024-03-29T11:28:56.615 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.624+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f25f00837d0 con 0x7f26001457f0 2024-03-29T11:28:56.615 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.624+0000 7f25f67fc640 10 client.4510.objecter ms_dispatch 0x7f2600000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:56.615 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.624+0000 7f25f67fc640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:56.615 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.624+0000 7f25f67fc640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:56.615 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.624+0000 7f25f67fc640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-29T11:28:56.617 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.627+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25f004f0b0 con 0x7f26001457f0 2024-03-29T11:28:56.617 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.627+0000 7f25f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:56.618 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.627+0000 7f25f67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:56.968 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.978+0000 7f2607d96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:56.968 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.978+0000 7f2607d96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:56.968 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.978+0000 7f2607d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:28:56.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.978+0000 7f2607d96640 1 -- 172.21.15.67:0/2673048656 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25e8005330 con 0x7f26001457f0 2024-03-29T11:28:56.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.979+0000 7f25f67fc640 1 -- 172.21.15.67:0/2673048656 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f25f0050fe0 con 0x7f26001457f0 2024-03-29T11:28:56.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.979+0000 7f25f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:56.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.979+0000 7f25f67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:56.969 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:28:56.970 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:56.972 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.982+0000 7f2607d96640 10 librados: watch_flush enter 2024-03-29T11:28:56.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.983+0000 7f2607d96640 10 librados: watch_flush exit 2024-03-29T11:28:56.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.983+0000 7f2607d96640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-29T11:28:56.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.983+0000 7f2607d96640 10 client.4510.objecter successfully canceled tick 2024-03-29T11:28:56.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.983+0000 7f2607d96640 1 -- 172.21.15.67:0/2673048656 >> v1:172.21.15.67:6816/404787197 conn(0x7f25e406de50 legacy=0x7f25e4070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.983+0000 7f2607d96640 10 monclient: shutdown 2024-03-29T11:28:56.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.983+0000 7f2607d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:56.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.983+0000 7f2607d96640 1 -- 172.21.15.67:0/2673048656 >> v1:172.21.15.82:6789/0 conn(0x7f26001457f0 legacy=0x7f26001390a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.983+0000 7f2607d96640 1 -- 172.21.15.67:0/2673048656 shutdown_connections 2024-03-29T11:28:56.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.983+0000 7f2607d96640 1 -- 172.21.15.67:0/2673048656 >> 172.21.15.67:0/2673048656 conn(0x7f2600066a00 msgr2=0x7f2600066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:56.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.983+0000 7f2607d96640 1 -- 172.21.15.67:0/2673048656 shutdown_connections 2024-03-29T11:28:56.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.984+0000 7f2607d96640 1 -- 172.21.15.67:0/2673048656 wait complete. 2024-03-29T11:28:56.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:28:56.984+0000 7f2607d96640 1 librados: shutdown 2024-03-29T11:28:56.994 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:03.802 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:03.971 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.978+0000 7fb4a8ac9640 10 monclient: get_monmap_and_config 2024-03-29T11:29:03.971 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.978+0000 7fb4a8ac9640 10 monclient: build_initial_monmap 2024-03-29T11:29:03.971 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.978+0000 7fb4a8ac9640 10 monclient: monmap: 2024-03-29T11:29:03.972 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:03.972 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:03.972 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:29:03.979630+0000 2024-03-29T11:29:03.972 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:29:03.979630+0000 2024-03-29T11:29:03.972 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:03.972 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:03.972 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:03.972 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:29:03.972 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:29:03.973 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:03.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.980+0000 7fb4a8ac9640 1 Processor -- start 2024-03-29T11:29:03.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.980+0000 7fb4a8ac9640 1 -- start start 2024-03-29T11:29:03.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.980+0000 7fb4a8ac9640 10 monclient: init 2024-03-29T11:29:03.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.981+0000 7fb4a8ac9640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:03.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.981+0000 7fb4a8ac9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:03.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.981+0000 7fb4a8ac9640 10 monclient(hunting): picked mon.noname-a con 0x7fb4a4153210 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:03.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.981+0000 7fb4a8ac9640 10 monclient(hunting): picked mon.noname-b con 0x7fb4a40ab360 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:03.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.981+0000 7fb4a8ac9640 10 monclient(hunting): picked mon.noname-c con 0x7fb4a40ad6c0 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:03.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.982+0000 7fb4a8ac9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a40a94e0 con 0x7fb4a4153210 2024-03-29T11:29:03.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.982+0000 7fb4a8ac9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a40af3a0 con 0x7fb4a40ad6c0 2024-03-29T11:29:03.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.982+0000 7fb4a8ac9640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a40afb60 con 0x7fb4a40ab360 2024-03-29T11:29:03.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.982+0000 7fb4a8ac9640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:03.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.982+0000 7fb4a8ac9640 10 monclient(hunting): authenticate will time out at 1404.277986s 2024-03-29T11:29:03.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.982+0000 7fb4a259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb4a4153210 0x7fb4a414fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33134/0 (socket says 172.21.15.67:33134) 2024-03-29T11:29:03.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.982+0000 7fb4a259c640 1 -- 172.21.15.67:0/1317306814 learned_addr learned my addr 172.21.15.67:0/1317306814 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:03.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 1 -- 172.21.15.67:0/1317306814 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042479405 0 0) 0x7fb4a40a94e0 con 0x7fb4a4153210 2024-03-29T11:29:03.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:03.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:03.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:03.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:29:03.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 1 -- 172.21.15.67:0/1317306814 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c003520 con 0x7fb4a4153210 2024-03-29T11:29:03.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 1 -- 172.21.15.67:0/1317306814 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758519605 0 0) 0x7fb4a40af3a0 con 0x7fb4a40ad6c0 2024-03-29T11:29:03.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:03.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:03.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:03.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:29:03.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 1 -- 172.21.15.67:0/1317306814 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4a40a94e0 con 0x7fb4a40ad6c0 2024-03-29T11:29:03.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 1 -- 172.21.15.67:0/1317306814 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 830097363 0 0) 0x7fb48c003520 con 0x7fb4a4153210 2024-03-29T11:29:03.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 1 -- 172.21.15.67:0/1317306814 >> v1:172.21.15.67:6790/0 conn(0x7fb4a40ab360 legacy=0x7fb4a41578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.983+0000 7fb4a159a640 1 -- 172.21.15.67:0/1317306814 >> v1:172.21.15.82:6789/0 conn(0x7fb4a40ad6c0 legacy=0x7fb4a415aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.984+0000 7fb4a159a640 10 monclient: _finish_hunting 0 2024-03-29T11:29:03.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.984+0000 7fb4a159a640 1 monclient: found mon.noname-a 2024-03-29T11:29:03.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.984+0000 7fb4a159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:03.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.984+0000 7fb4a159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:29:03.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.984+0000 7fb4a159a640 1 -- 172.21.15.67:0/1317306814 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a40aab80 con 0x7fb4a4153210 2024-03-29T11:29:03.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.984+0000 7fb4a159a640 10 monclient: _finish_auth 0 2024-03-29T11:29:03.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.984+0000 7fb4a159a640 10 monclient: _check_auth_tickets 2024-03-29T11:29:03.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.984+0000 7fb4a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:03.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.984+0000 7fb4a159a640 1 -- 172.21.15.67:0/1317306814 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb49c001690 con 0x7fb4a4153210 2024-03-29T11:29:03.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.984+0000 7fb4a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:03.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.984+0000 7fb4a159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:03.978 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.984+0000 7fb4a159a640 10 monclient: dump: 2024-03-29T11:29:03.978 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:03.978 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:03.978 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:03.978 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:03.978 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:03.978 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:03.978 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:03.978 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:03.978 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:03.978 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:03.978 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.984+0000 7fb4a8ac9640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:29:03.978 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.984+0000 7fb4a8ac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:03.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.985+0000 7fb4a159a640 1 -- 172.21.15.67:0/1317306814 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb49c0017f0 con 0x7fb4a4153210 2024-03-29T11:29:03.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.985+0000 7fb4a159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:03.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.985+0000 7fb4a159a640 1 -- 172.21.15.67:0/1317306814 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb49c001c40 con 0x7fb4a4153210 2024-03-29T11:29:03.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.985+0000 7fb4a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:03.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.985+0000 7fb4a159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:03.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.985+0000 7fb4a159a640 10 monclient: dump: 2024-03-29T11:29:03.980 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:03.980 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:03.980 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:03.980 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:03.980 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:03.980 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.985+0000 7fb4a8ac9640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.985+0000 7fb4a8ac9640 4 set_mon_vals no callback set 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.986+0000 7fb4a8ac9640 10 monclient: shutdown 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.986+0000 7fb4a8ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.986+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/1317306814 >> v1:172.21.15.67:6789/0 conn(0x7fb4a4153210 legacy=0x7fb4a414fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.986+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/1317306814 shutdown_connections 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.986+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/1317306814 >> 172.21.15.67:0/1317306814 conn(0x7fb4a40b35e0 msgr2=0x7fb4a40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.986+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/1317306814 shutdown_connections 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.986+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/1317306814 wait complete. 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.987+0000 7fb4a8ac9640 10 monclient: build_initial_monmap 2024-03-29T11:29:03.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.987+0000 7fb4a8ac9640 10 monclient: monmap: 2024-03-29T11:29:03.982 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:03.982 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:03.982 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:29:03.982 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:29:03.982 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:03.982 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:03.982 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:03.982 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:29:03.982 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:29:03.982 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:03.982 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.988+0000 7fb4a8ac9640 1 librados: starting msgr at 2024-03-29T11:29:03.982 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.988+0000 7fb4a8ac9640 1 librados: starting objecter 2024-03-29T11:29:03.983 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.988+0000 7fb4a8ac9640 1 Processor -- start 2024-03-29T11:29:03.983 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.988+0000 7fb4a8ac9640 1 -- start start 2024-03-29T11:29:03.983 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.988+0000 7fb4a8ac9640 1 librados: setting wanted keys 2024-03-29T11:29:03.983 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.988+0000 7fb4a8ac9640 1 librados: calling monclient init 2024-03-29T11:29:03.983 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.988+0000 7fb4a8ac9640 10 monclient: init 2024-03-29T11:29:03.983 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.989+0000 7fb4a8ac9640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:03.983 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.989+0000 7fb4a8ac9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:03.983 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.989+0000 7fb4a8ac9640 10 monclient(hunting): picked mon.noname-a con 0x7fb4a40ab360 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:03.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.990+0000 7fb4a8ac9640 10 monclient(hunting): picked mon.noname-c con 0x7fb4a40ad6c0 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:03.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.990+0000 7fb4a8ac9640 10 monclient(hunting): picked mon.noname-b con 0x7fb4a4153210 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:03.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.990+0000 7fb4a8ac9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a40708a0 con 0x7fb4a40ab360 2024-03-29T11:29:03.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.990+0000 7fb4a8ac9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a4071890 con 0x7fb4a4153210 2024-03-29T11:29:03.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.990+0000 7fb4a8ac9640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a4071c40 con 0x7fb4a40ad6c0 2024-03-29T11:29:03.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.990+0000 7fb4a8ac9640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:03.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.990+0000 7fb4a8ac9640 10 monclient(hunting): authenticate will time out at 1404.286286s 2024-03-29T11:29:03.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.990+0000 7fb4a2d9d640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fb4a4153210 0x7fb4a41f7df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:56822/0 (socket says 172.21.15.67:56822) 2024-03-29T11:29:03.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.990+0000 7fb4a2d9d640 1 -- 172.21.15.67:0/666845571 learned_addr learned my addr 172.21.15.67:0/666845571 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:03.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.990+0000 7fb492ffd640 10 client.?.objecter ms_handle_connect 0x7fb4a40ad6c0 2024-03-29T11:29:03.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.990+0000 7fb492ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:03.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.990+0000 7fb492ffd640 10 client.?.objecter ms_handle_connect 0x7fb4a4153210 2024-03-29T11:29:03.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.990+0000 7fb492ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:03.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.991+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607482978 0 0) 0x7fb4a4071890 con 0x7fb4a4153210 2024-03-29T11:29:03.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.991+0000 7fb492ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:03.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.991+0000 7fb492ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:03.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.991+0000 7fb492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:03.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.991+0000 7fb492ffd640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:29:03.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.991+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb484004eb0 con 0x7fb4a4153210 2024-03-29T11:29:03.986 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.991+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198432599 0 0) 0x7fb4a4071c40 con 0x7fb4a40ad6c0 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.991+0000 7fb492ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.991+0000 7fb492ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.991+0000 7fb492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.991+0000 7fb492ffd640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.991+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4a4071890 con 0x7fb4a40ad6c0 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.991+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1409639557 0 0) 0x7fb484004eb0 con 0x7fb4a4153210 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.991+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4a4071c40 con 0x7fb4a4153210 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb494001cc0 con 0x7fb4a4153210 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:03.987 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3273636828 0 0) 0x7fb4a4071c40 con 0x7fb4a4153210 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 >> v1:172.21.15.67:6790/0 conn(0x7fb4a40ad6c0 legacy=0x7fb4a408a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 >> v1:172.21.15.67:6789/0 conn(0x7fb4a40ab360 legacy=0x7fb4a406fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 1 monclient: found mon.b 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a41ea010 con 0x7fb4a4153210 2024-03-29T11:29:03.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.992+0000 7fb492ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.993+0000 7fb4a8ac9640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.993+0000 7fb4a8ac9640 10 monclient: _renew_subs 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.993+0000 7fb4a8ac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.993+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/666845571 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a4072490 con 0x7fb4a4153210 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.993+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb494001e20 con 0x7fb4a4153210 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.993+0000 7fb4a8ac9640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.993+0000 7fb4a8ac9640 10 monclient: _renew_subs 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.993+0000 7fb4a8ac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.993+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/666845571 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a40705e0 con 0x7fb4a4153210 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.993+0000 7fb4a8ac9640 1 librados: init done 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.993+0000 7fb492ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.993+0000 7fb4a0d99640 4 set_mon_vals no callback set 2024-03-29T11:29:03.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.994+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fb494003dc0 con 0x7fb4a4153210 2024-03-29T11:29:03.990 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.994+0000 7fb492ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:03.990 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.994+0000 7fb492ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:03.990 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.994+0000 7fb492ffd640 10 monclient: dump: 2024-03-29T11:29:03.990 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:03.990 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:03.990 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:03.990 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:03.990 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:03.990 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:03.990 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:03.990 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:03.990 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:03.990 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:03.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.996+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fb494003f20 con 0x7fb4a4153210 2024-03-29T11:29:03.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.997+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fb494084df0 con 0x7fb4a4153210 2024-03-29T11:29:03.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.997+0000 7fb492ffd640 10 client.4516.objecter ms_dispatch 0x7fb4a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:03.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.997+0000 7fb4a259c640 10 monclient: get_auth_request con 0x7fb48406dce0 auth_method 0 2024-03-29T11:29:03.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.997+0000 7fb492ffd640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:03.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.997+0000 7fb492ffd640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:03.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.997+0000 7fb492ffd640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-29T11:29:03.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.997+0000 7fb492ffd640 10 client.4516.objecter ms_handle_connect 0x7fb48406dce0 2024-03-29T11:29:03.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.998+0000 7fb4a8ac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:03.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.998+0000 7fb4a8ac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:03.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.998+0000 7fb4a8ac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:03.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:03.998+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/666845571 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb488005080 con 0x7fb4a4153210 2024-03-29T11:29:03.997 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.006+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb494050500 con 0x7fb4a4153210 2024-03-29T11:29:03.997 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.006+0000 7fb492ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:03.997 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.006+0000 7fb492ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:04.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.388+0000 7fb4a8ac9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:04.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.388+0000 7fb4a8ac9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:04.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.388+0000 7fb4a8ac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:04.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.388+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/666845571 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb488005330 con 0x7fb4a4153210 2024-03-29T11:29:04.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.389+0000 7fb492ffd640 1 -- 172.21.15.67:0/666845571 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fb494052600 con 0x7fb4a4153210 2024-03-29T11:29:04.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.389+0000 7fb492ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:04.381 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.389+0000 7fb492ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:04.382 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:29:04.382 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:04.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.395+0000 7fb4a8ac9640 10 librados: watch_flush enter 2024-03-29T11:29:04.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.395+0000 7fb4a8ac9640 10 librados: watch_flush exit 2024-03-29T11:29:04.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.395+0000 7fb4a8ac9640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-29T11:29:04.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.395+0000 7fb4a8ac9640 10 client.4516.objecter successfully canceled tick 2024-03-29T11:29:04.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.396+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/666845571 >> v1:172.21.15.67:6816/404787197 conn(0x7fb48406dce0 legacy=0x7fb4840701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.396+0000 7fb4a8ac9640 10 monclient: shutdown 2024-03-29T11:29:04.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.396+0000 7fb4a8ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:04.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.396+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/666845571 >> v1:172.21.15.82:6789/0 conn(0x7fb4a4153210 legacy=0x7fb4a41f7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.398+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/666845571 shutdown_connections 2024-03-29T11:29:04.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.398+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/666845571 >> 172.21.15.67:0/666845571 conn(0x7fb4a40b35e0 msgr2=0x7fb4a40b1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:04.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.399+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/666845571 shutdown_connections 2024-03-29T11:29:04.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.399+0000 7fb4a8ac9640 1 -- 172.21.15.67:0/666845571 wait complete. 2024-03-29T11:29:04.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:04.399+0000 7fb4a8ac9640 1 librados: shutdown 2024-03-29T11:29:04.407 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:11.215 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:11.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.377+0000 7f5ec3e93640 10 monclient: get_monmap_and_config 2024-03-29T11:29:11.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.377+0000 7f5ec3e93640 10 monclient: build_initial_monmap 2024-03-29T11:29:11.368 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.377+0000 7f5ec3e93640 10 monclient: monmap: 2024-03-29T11:29:11.368 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:11.368 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:11.369 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:29:11.378448+0000 2024-03-29T11:29:11.369 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:29:11.378448+0000 2024-03-29T11:29:11.369 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:11.369 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:11.369 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:11.369 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:29:11.369 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:29:11.369 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:11.370 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.378+0000 7f5ec3e93640 1 Processor -- start 2024-03-29T11:29:11.370 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.378+0000 7f5ec3e93640 1 -- start start 2024-03-29T11:29:11.370 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.378+0000 7f5ec3e93640 10 monclient: init 2024-03-29T11:29:11.370 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.379+0000 7f5ec3e93640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:11.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.379+0000 7f5ec3e93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:11.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.379+0000 7f5ec3e93640 10 monclient(hunting): picked mon.noname-c con 0x7f5ebc0b46b0 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:11.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.379+0000 7f5ec3e93640 10 monclient(hunting): picked mon.noname-a con 0x7f5ebc0aeae0 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:11.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.379+0000 7f5ec3e93640 10 monclient(hunting): picked mon.noname-b con 0x7f5ebc13fa10 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:11.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.379+0000 7f5ec3e93640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc0b06a0 con 0x7f5ebc0aeae0 2024-03-29T11:29:11.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.379+0000 7f5ec3e93640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc0b4230 con 0x7f5ebc0b46b0 2024-03-29T11:29:11.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.379+0000 7f5ec3e93640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc0b0910 con 0x7f5ebc13fa10 2024-03-29T11:29:11.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.379+0000 7f5ec3e93640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:11.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.379+0000 7f5ec3e93640 10 monclient(hunting): authenticate will time out at 1411.675799s 2024-03-29T11:29:11.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.380+0000 7f5ec2430640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f5ebc13fa10 0x7f5ebc14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:52730/0 (socket says 172.21.15.67:52730) 2024-03-29T11:29:11.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.380+0000 7f5ec2430640 1 -- 172.21.15.67:0/3750742735 learned_addr learned my addr 172.21.15.67:0/3750742735 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:11.371 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.380+0000 7f5ec1c2f640 1 -- 172.21.15.67:0/3750742735 >> v1:172.21.15.82:6789/0 conn(0x7f5ebc0b46b0 legacy=0x7f5ebc0b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-29T11:29:11.372 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.380+0000 7f5ec1c2f640 1 -- 172.21.15.67:0/3750742735 >> v1:172.21.15.82:6789/0 conn(0x7f5ebc0b46b0 legacy=0x7f5ebc0b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T11:29:11.372 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.380+0000 7f5ec1c2f640 1 --1- 172.21.15.67:0/3750742735 >> v1:172.21.15.82:6789/0 conn(0x7f5ebc0b46b0 0x7f5ebc0b3b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-29T11:29:11.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.380+0000 7f5ec0c2d640 1 -- 172.21.15.67:0/3750742735 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342508788 0 0) 0x7f5ebc0b0910 con 0x7f5ebc13fa10 2024-03-29T11:29:11.374 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.380+0000 7f5ec0c2d640 10 monclient(hunting): _negotiate 2024-03-29T11:29:11.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.380+0000 7f5ec0c2d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:11.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.380+0000 7f5ec0c2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:11.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.380+0000 7f5ec0c2d640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:29:11.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.380+0000 7f5ec0c2d640 1 -- 172.21.15.67:0/3750742735 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea8003520 con 0x7f5ebc13fa10 2024-03-29T11:29:11.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 1 -- 172.21.15.67:0/3750742735 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699069297 0 0) 0x7f5ebc0b06a0 con 0x7f5ebc0aeae0 2024-03-29T11:29:11.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 10 monclient(hunting): _negotiate 2024-03-29T11:29:11.375 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:11.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:11.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:29:11.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 1 -- 172.21.15.67:0/3750742735 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ebc0b0910 con 0x7f5ebc0aeae0 2024-03-29T11:29:11.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 1 -- 172.21.15.67:0/3750742735 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3126106791 0 0) 0x7f5ea8003520 con 0x7f5ebc13fa10 2024-03-29T11:29:11.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 1 -- 172.21.15.67:0/3750742735 >> v1:172.21.15.82:6789/0 conn(0x7f5ebc0b46b0 legacy=0x7f5ebc0b3b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:11.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 1 -- 172.21.15.67:0/3750742735 >> v1:172.21.15.67:6789/0 conn(0x7f5ebc0aeae0 legacy=0x7f5ebc13f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 10 monclient: _finish_hunting 0 2024-03-29T11:29:11.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 1 monclient: found mon.noname-b 2024-03-29T11:29:11.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:11.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:29:11.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 1 -- 172.21.15.67:0/3750742735 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc0af6d0 con 0x7f5ebc13fa10 2024-03-29T11:29:11.376 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 10 monclient: _finish_auth 0 2024-03-29T11:29:11.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 10 monclient: _check_auth_tickets 2024-03-29T11:29:11.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:11.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec3e93640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:29:11.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 1 -- 172.21.15.67:0/3750742735 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5eb4001a10 con 0x7f5ebc13fa10 2024-03-29T11:29:11.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:11.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:11.377 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec0c2d640 10 monclient: dump: 2024-03-29T11:29:11.377 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:11.377 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:11.377 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:11.377 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:11.377 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:11.377 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:11.378 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:11.378 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:11.378 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:11.378 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:11.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.381+0000 7f5ec3e93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:11.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.382+0000 7f5ec0c2d640 1 -- 172.21.15.67:0/3750742735 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5eb4001c30 con 0x7f5ebc13fa10 2024-03-29T11:29:11.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.382+0000 7f5ec0c2d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:11.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.382+0000 7f5ec0c2d640 1 -- 172.21.15.67:0/3750742735 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5eb4001fc0 con 0x7f5ebc13fa10 2024-03-29T11:29:11.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.382+0000 7f5ec0c2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:11.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.382+0000 7f5ec0c2d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:11.378 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.382+0000 7f5ec0c2d640 10 monclient: dump: 2024-03-29T11:29:11.378 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:11.378 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:11.378 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:11.379 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:11.379 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:11.379 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:11.379 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:11.379 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:11.379 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:11.379 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:11.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.382+0000 7f5ec3e93640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:11.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.382+0000 7f5ec3e93640 4 set_mon_vals no callback set 2024-03-29T11:29:11.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.382+0000 7f5ec3e93640 10 monclient: shutdown 2024-03-29T11:29:11.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.382+0000 7f5ec3e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:11.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.382+0000 7f5ec3e93640 1 -- 172.21.15.67:0/3750742735 >> v1:172.21.15.67:6790/0 conn(0x7f5ebc13fa10 legacy=0x7f5ebc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.382+0000 7f5ec3e93640 1 -- 172.21.15.67:0/3750742735 shutdown_connections 2024-03-29T11:29:11.379 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.382+0000 7f5ec3e93640 1 -- 172.21.15.67:0/3750742735 >> 172.21.15.67:0/3750742735 conn(0x7f5ebc066a00 msgr2=0x7f5ebc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:11.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.383+0000 7f5ec3e93640 1 -- 172.21.15.67:0/3750742735 shutdown_connections 2024-03-29T11:29:11.380 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.383+0000 7f5ec3e93640 1 -- 172.21.15.67:0/3750742735 wait complete. 2024-03-29T11:29:11.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.384+0000 7f5ec3e93640 10 monclient: build_initial_monmap 2024-03-29T11:29:11.383 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.384+0000 7f5ec3e93640 10 monclient: monmap: 2024-03-29T11:29:11.383 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:11.383 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:11.383 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:29:11.383 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:29:11.383 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:11.383 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:11.383 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:11.384 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:29:11.384 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:29:11.384 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:11.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.384+0000 7f5ec3e93640 1 librados: starting msgr at 2024-03-29T11:29:11.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.384+0000 7f5ec3e93640 1 librados: starting objecter 2024-03-29T11:29:11.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.385+0000 7f5ec3e93640 1 Processor -- start 2024-03-29T11:29:11.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.385+0000 7f5ec3e93640 1 -- start start 2024-03-29T11:29:11.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.385+0000 7f5ec3e93640 1 librados: setting wanted keys 2024-03-29T11:29:11.384 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.385+0000 7f5ec3e93640 1 librados: calling monclient init 2024-03-29T11:29:11.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.385+0000 7f5ec3e93640 10 monclient: init 2024-03-29T11:29:11.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.385+0000 7f5ec3e93640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:11.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.385+0000 7f5ec3e93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:11.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.385+0000 7f5ec3e93640 10 monclient(hunting): picked mon.noname-a con 0x7f5ebc0aeae0 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:11.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.385+0000 7f5ec3e93640 10 monclient(hunting): picked mon.noname-c con 0x7f5ebc0b46b0 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:11.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5ec3e93640 10 monclient(hunting): picked mon.noname-b con 0x7f5ebc13fa10 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:11.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5ec3e93640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc140d90 con 0x7f5ebc0aeae0 2024-03-29T11:29:11.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5ec3e93640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc141000 con 0x7f5ebc13fa10 2024-03-29T11:29:11.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5ec3e93640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc145de0 con 0x7f5ebc0b46b0 2024-03-29T11:29:11.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5ec3e93640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:11.385 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5ec3e93640 10 monclient(hunting): authenticate will time out at 1411.682088s 2024-03-29T11:29:11.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5ec2430640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f5ebc13fa10 0x7f5ebc1e2a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:60642/0 (socket says 172.21.15.67:60642) 2024-03-29T11:29:11.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5ec2430640 1 -- 172.21.15.67:0/1227098340 learned_addr learned my addr 172.21.15.67:0/1227098340 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:11.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5eb27fc640 10 client.?.objecter ms_handle_connect 0x7f5ebc13fa10 2024-03-29T11:29:11.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5eb27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:11.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5eb27fc640 10 client.?.objecter ms_handle_connect 0x7f5ebc0b46b0 2024-03-29T11:29:11.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5eb27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:11.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262888892 0 0) 0x7f5ebc141000 con 0x7f5ebc13fa10 2024-03-29T11:29:11.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5eb27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:11.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5eb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:11.386 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5eb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:11.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5eb27fc640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:29:11.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea0004eb0 con 0x7f5ebc13fa10 2024-03-29T11:29:11.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5eb27fc640 10 client.?.objecter ms_handle_connect 0x7f5ebc0aeae0 2024-03-29T11:29:11.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.386+0000 7f5eb27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:11.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1032501840 0 0) 0x7f5ea0004eb0 con 0x7f5ebc13fa10 2024-03-29T11:29:11.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ebc141000 con 0x7f5ebc13fa10 2024-03-29T11:29:11.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5eb4001f40 con 0x7f5ebc13fa10 2024-03-29T11:29:11.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:11.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:11.387 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient(hunting): dump: 2024-03-29T11:29:11.388 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:11.388 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:11.388 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:11.388 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:11.388 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:11.388 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:11.388 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:11.388 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:11.388 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:11.388 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:11.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:11.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893298943 0 0) 0x7f5ebc140d90 con 0x7f5ebc0aeae0 2024-03-29T11:29:11.388 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:11.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:11.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:11.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:29:11.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea0004eb0 con 0x7f5ebc0aeae0 2024-03-29T11:29:11.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751950489 0 0) 0x7f5ebc145de0 con 0x7f5ebc0b46b0 2024-03-29T11:29:11.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:11.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:11.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:11.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:29:11.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ebc140d90 con 0x7f5ebc0b46b0 2024-03-29T11:29:11.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3389116360 0 0) 0x7f5ebc141000 con 0x7f5ebc13fa10 2024-03-29T11:29:11.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 >> v1:172.21.15.67:6790/0 conn(0x7f5ebc0b46b0 legacy=0x7f5ebc1da690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 >> v1:172.21.15.67:6789/0 conn(0x7f5ebc0aeae0 legacy=0x7f5ebc1427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.389 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:11.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 1 monclient: found mon.b 2024-03-29T11:29:11.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:11.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:11.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc146300 con 0x7f5ebc13fa10 2024-03-29T11:29:11.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:11.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:11.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:11.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5ec3e93640 5 monclient: authenticate success, global_id 4519 2024-03-29T11:29:11.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5ec3e93640 10 monclient: _renew_subs 2024-03-29T11:29:11.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5ec3e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:11.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5ec3e93640 1 -- 172.21.15.67:0/1227098340 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ebc145a00 con 0x7f5ebc13fa10 2024-03-29T11:29:11.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5ec3e93640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:11.390 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5ec3e93640 10 monclient: _renew_subs 2024-03-29T11:29:11.391 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5ec3e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:11.391 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5ec3e93640 1 -- 172.21.15.67:0/1227098340 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ebc145720 con 0x7f5ebc13fa10 2024-03-29T11:29:11.391 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5ec3e93640 1 librados: init done 2024-03-29T11:29:11.391 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5eb40020a0 con 0x7f5ebc13fa10 2024-03-29T11:29:11.391 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:11.391 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.387+0000 7f5eb37fe640 4 set_mon_vals no callback set 2024-03-29T11:29:11.391 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.388+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f5eb4004fb0 con 0x7f5ebc13fa10 2024-03-29T11:29:11.391 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.388+0000 7f5eb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:11.391 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.388+0000 7f5eb27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:11.391 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.388+0000 7f5eb27fc640 10 monclient: dump: 2024-03-29T11:29:11.391 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:11.391 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:11.391 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:11.391 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:11.392 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:11.392 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:11.392 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:11.392 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:11.392 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:11.392 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:11.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.390+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f5eb401af80 con 0x7f5ebc13fa10 2024-03-29T11:29:11.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.390+0000 7f5ec3e93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:11.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.390+0000 7f5ec3e93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:11.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.390+0000 7f5ec3e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:11.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.390+0000 7f5ec3e93640 1 -- 172.21.15.67:0/1227098340 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea4005080 con 0x7f5ebc13fa10 2024-03-29T11:29:11.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.391+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f5eb4084f70 con 0x7f5ebc13fa10 2024-03-29T11:29:11.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.391+0000 7f5eb27fc640 10 client.4519.objecter ms_dispatch 0x7f5ebc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:11.392 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.391+0000 7f5eb27fc640 3 client.4519.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:11.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.391+0000 7f5eb27fc640 3 client.4519.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:11.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.391+0000 7f5ec1c2f640 10 monclient: get_auth_request con 0x7f5ea006de50 auth_method 0 2024-03-29T11:29:11.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.391+0000 7f5eb27fc640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-29T11:29:11.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.391+0000 7f5eb27fc640 10 client.4519.objecter ms_handle_connect 0x7f5ea006de50 2024-03-29T11:29:11.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.399+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5eb4050850 con 0x7f5ebc13fa10 2024-03-29T11:29:11.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.399+0000 7f5eb27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:11.393 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.399+0000 7f5eb27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:11.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.794+0000 7f5ec3e93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:11.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.794+0000 7f5ec3e93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:11.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.794+0000 7f5ec3e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:11.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.794+0000 7f5ec3e93640 1 -- 172.21.15.67:0/1227098340 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ea4005330 con 0x7f5ebc13fa10 2024-03-29T11:29:11.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.796+0000 7f5eb27fc640 1 -- 172.21.15.67:0/1227098340 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f5eb4052780 con 0x7f5ebc13fa10 2024-03-29T11:29:11.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.796+0000 7f5eb27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:11.787 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.796+0000 7f5eb27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:11.787 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:29:11.788 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:11.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.800+0000 7f5ec3e93640 10 librados: watch_flush enter 2024-03-29T11:29:11.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.800+0000 7f5ec3e93640 10 librados: watch_flush exit 2024-03-29T11:29:11.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.800+0000 7f5ec3e93640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-29T11:29:11.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.800+0000 7f5ec3e93640 10 client.4519.objecter successfully canceled tick 2024-03-29T11:29:11.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.800+0000 7f5ec3e93640 1 -- 172.21.15.67:0/1227098340 >> v1:172.21.15.67:6816/404787197 conn(0x7f5ea006de50 legacy=0x7f5ea0070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.800+0000 7f5ec3e93640 10 monclient: shutdown 2024-03-29T11:29:11.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.800+0000 7f5ec3e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:11.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.800+0000 7f5ec3e93640 1 -- 172.21.15.67:0/1227098340 >> v1:172.21.15.82:6789/0 conn(0x7f5ebc13fa10 legacy=0x7f5ebc1e2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.800+0000 7f5ec3e93640 1 -- 172.21.15.67:0/1227098340 shutdown_connections 2024-03-29T11:29:11.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.800+0000 7f5ec3e93640 1 -- 172.21.15.67:0/1227098340 >> 172.21.15.67:0/1227098340 conn(0x7f5ebc066a00 msgr2=0x7f5ebc13a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:11.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.801+0000 7f5ec3e93640 1 -- 172.21.15.67:0/1227098340 shutdown_connections 2024-03-29T11:29:11.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.801+0000 7f5ec3e93640 1 -- 172.21.15.67:0/1227098340 wait complete. 2024-03-29T11:29:11.793 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:11.801+0000 7f5ec3e93640 1 librados: shutdown 2024-03-29T11:29:11.809 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:18.616 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:18.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.782+0000 7fadd1c33640 10 monclient: get_monmap_and_config 2024-03-29T11:29:18.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.782+0000 7fadd1c33640 10 monclient: build_initial_monmap 2024-03-29T11:29:18.774 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.782+0000 7fadd1c33640 10 monclient: monmap: 2024-03-29T11:29:18.774 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:18.774 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:18.774 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:29:18.783416+0000 2024-03-29T11:29:18.775 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:29:18.783416+0000 2024-03-29T11:29:18.775 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:18.775 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:18.775 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:18.775 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:29:18.775 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:29:18.775 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:18.776 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.784+0000 7fadd1c33640 1 Processor -- start 2024-03-29T11:29:18.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.784+0000 7fadd1c33640 1 -- start start 2024-03-29T11:29:18.777 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.784+0000 7fadd1c33640 10 monclient: init 2024-03-29T11:29:18.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.786+0000 7fadd1c33640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:18.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.786+0000 7fadd1c33640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:18.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.786+0000 7fadd1c33640 10 monclient(hunting): picked mon.noname-a con 0x7fadcc0a9930 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:18.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.786+0000 7fadd1c33640 10 monclient(hunting): picked mon.noname-b con 0x7fadcc149dc0 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:18.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.787+0000 7fadd1c33640 10 monclient(hunting): picked mon.noname-c con 0x7fadcc14ff40 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:18.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.787+0000 7fadd1c33640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadcc0aae60 con 0x7fadcc0a9930 2024-03-29T11:29:18.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.787+0000 7fadd1c33640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadcc0b4310 con 0x7fadcc14ff40 2024-03-29T11:29:18.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.787+0000 7fadd1c33640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadcc0afdd0 con 0x7fadcc149dc0 2024-03-29T11:29:18.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.787+0000 7fadd1c33640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:18.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.787+0000 7fadd1c33640 10 monclient(hunting): authenticate will time out at 1419.083042s 2024-03-29T11:29:18.778 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.787+0000 7fadcb7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fadcc0a9930 0x7fadcc143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49154/0 (socket says 172.21.15.67:49154) 2024-03-29T11:29:18.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.787+0000 7fadcb7fe640 1 -- 172.21.15.67:0/3945794108 learned_addr learned my addr 172.21.15.67:0/3945794108 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:18.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.787+0000 7fadca7fc640 1 -- 172.21.15.67:0/3945794108 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288211455 0 0) 0x7fadcc0aae60 con 0x7fadcc0a9930 2024-03-29T11:29:18.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.787+0000 7fadca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:18.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.787+0000 7fadca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:18.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.787+0000 7fadca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:18.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.787+0000 7fadca7fc640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:29:18.779 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.787+0000 7fadca7fc640 1 -- 172.21.15.67:0/3945794108 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadb4003520 con 0x7fadcc0a9930 2024-03-29T11:29:18.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.790+0000 7fadca7fc640 1 -- 172.21.15.67:0/3945794108 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3393504297 0 0) 0x7fadb4003520 con 0x7fadcc0a9930 2024-03-29T11:29:18.781 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.790+0000 7fadca7fc640 1 -- 172.21.15.67:0/3945794108 >> v1:172.21.15.67:6790/0 conn(0x7fadcc149dc0 legacy=0x7fadcc14a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:18.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.791+0000 7fadca7fc640 1 -- 172.21.15.67:0/3945794108 >> v1:172.21.15.82:6789/0 conn(0x7fadcc14ff40 legacy=0x7fadcc152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:18.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.791+0000 7fadca7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:18.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.791+0000 7fadca7fc640 1 monclient: found mon.noname-a 2024-03-29T11:29:18.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.791+0000 7fadca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:18.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.791+0000 7fadca7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:29:18.784 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.791+0000 7fadca7fc640 1 -- 172.21.15.67:0/3945794108 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc14a950 con 0x7fadcc0a9930 2024-03-29T11:29:18.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.791+0000 7fadca7fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:18.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.791+0000 7fadca7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:18.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.791+0000 7fadca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:18.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.791+0000 7fadca7fc640 1 -- 172.21.15.67:0/3945794108 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fadbc001690 con 0x7fadcc0a9930 2024-03-29T11:29:18.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.791+0000 7fadd1c33640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:29:18.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.792+0000 7fadd1c33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:18.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.792+0000 7fadca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:18.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.792+0000 7fadca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:18.785 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.792+0000 7fadca7fc640 10 monclient: dump: 2024-03-29T11:29:18.785 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:18.785 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:18.785 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:18.785 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:18.786 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:18.786 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:18.786 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:18.786 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:18.786 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:18.786 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:18.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.792+0000 7fadca7fc640 1 -- 172.21.15.67:0/3945794108 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadbc001920 con 0x7fadcc0a9930 2024-03-29T11:29:18.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.792+0000 7fadd1c33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:18.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.792+0000 7fadca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:18.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.792+0000 7fadca7fc640 1 -- 172.21.15.67:0/3945794108 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fadbc001cf0 con 0x7fadcc0a9930 2024-03-29T11:29:18.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.792+0000 7fadd1c33640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:18.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.792+0000 7fadd1c33640 4 set_mon_vals no callback set 2024-03-29T11:29:18.786 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.792+0000 7fadca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.792+0000 7fadca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.792+0000 7fadca7fc640 10 monclient: dump: 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.793+0000 7fadd1c33640 10 monclient: shutdown 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.793+0000 7fadd1c33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:18.787 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.793+0000 7fadd1c33640 1 -- 172.21.15.67:0/3945794108 >> v1:172.21.15.67:6789/0 conn(0x7fadcc0a9930 legacy=0x7fadcc143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:18.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.793+0000 7fadd1c33640 1 -- 172.21.15.67:0/3945794108 shutdown_connections 2024-03-29T11:29:18.788 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.793+0000 7fadd1c33640 1 -- 172.21.15.67:0/3945794108 >> 172.21.15.67:0/3945794108 conn(0x7fadcc0b35e0 msgr2=0x7fadcc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:18.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.793+0000 7fadd1c33640 1 -- 172.21.15.67:0/3945794108 shutdown_connections 2024-03-29T11:29:18.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.793+0000 7fadd1c33640 1 -- 172.21.15.67:0/3945794108 wait complete. 2024-03-29T11:29:18.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.794+0000 7fadd1c33640 10 monclient: build_initial_monmap 2024-03-29T11:29:18.790 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.795+0000 7fadd1c33640 10 monclient: monmap: 2024-03-29T11:29:18.790 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:18.790 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:18.790 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.795+0000 7fadd1c33640 1 librados: starting msgr at 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.795+0000 7fadd1c33640 1 librados: starting objecter 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.795+0000 7fadd1c33640 1 Processor -- start 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.796+0000 7fadd1c33640 1 -- start start 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.796+0000 7fadd1c33640 1 librados: setting wanted keys 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.796+0000 7fadd1c33640 1 librados: calling monclient init 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.796+0000 7fadd1c33640 10 monclient: init 2024-03-29T11:29:18.791 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.797+0000 7fadd1c33640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:18.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.797+0000 7fadd1c33640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:18.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.797+0000 7fadd1c33640 10 monclient(hunting): picked mon.noname-c con 0x7fadcc149dc0 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:18.792 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.798+0000 7fadd1c33640 10 monclient(hunting): picked mon.noname-b con 0x7fadcc14ff40 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:18.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.800+0000 7fadd1c33640 10 monclient(hunting): picked mon.noname-a con 0x7fadcc06f110 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:18.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.800+0000 7fadd1c33640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadcc13fc20 con 0x7fadcc06f110 2024-03-29T11:29:18.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.800+0000 7fadd1c33640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadcc149a90 con 0x7fadcc14ff40 2024-03-29T11:29:18.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.800+0000 7fadd1c33640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadcc06f560 con 0x7fadcc149dc0 2024-03-29T11:29:18.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.800+0000 7fadd1c33640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:18.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.800+0000 7fadd1c33640 10 monclient(hunting): authenticate will time out at 1419.096692s 2024-03-29T11:29:18.796 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.800+0000 7fadcbfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fadcc06f110 0x7fadcc071b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52644/0 (socket says 172.21.15.67:52644) 2024-03-29T11:29:18.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.800+0000 7fadcbfff640 1 -- 172.21.15.67:0/527768068 learned_addr learned my addr 172.21.15.67:0/527768068 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:18.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.801+0000 7fadb3fff640 10 client.?.objecter ms_handle_connect 0x7fadcc06f110 2024-03-29T11:29:18.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.801+0000 7fadb3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:18.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.801+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440870755 0 0) 0x7fadcc13fc20 con 0x7fadcc06f110 2024-03-29T11:29:18.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.802+0000 7fadb3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:18.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.802+0000 7fadb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:18.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.802+0000 7fadb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:18.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.802+0000 7fadb3fff640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:29:18.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.802+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadb8004eb0 con 0x7fadcc06f110 2024-03-29T11:29:18.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.802+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2723485217 0 0) 0x7fadb8004eb0 con 0x7fadcc06f110 2024-03-29T11:29:18.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.802+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadcc13fc20 con 0x7fadcc06f110 2024-03-29T11:29:18.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.802+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fadc00017d0 con 0x7fadcc06f110 2024-03-29T11:29:18.797 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.802+0000 7fadb3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.802+0000 7fadb3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.802+0000 7fadb3fff640 10 monclient(hunting): dump: 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.802+0000 7fadb3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.802+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1647843739 0 0) 0x7fadcc13fc20 con 0x7fadcc06f110 2024-03-29T11:29:18.798 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.802+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 >> v1:172.21.15.67:6790/0 conn(0x7fadcc149dc0 legacy=0x7fadcc13f510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 >> v1:172.21.15.82:6789/0 conn(0x7fadcc14ff40 legacy=0x7fadcc1dec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadb3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadb3fff640 1 monclient: found mon.a 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadb3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc1df340 con 0x7fadcc06f110 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadb3fff640 10 monclient: _finish_auth 0 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadb3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadd1c33640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadd1c33640 10 monclient: _renew_subs 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadd1c33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadd1c33640 1 -- 172.21.15.67:0/527768068 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadcc0702a0 con 0x7fadcc06f110 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadd1c33640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadd1c33640 10 monclient: _renew_subs 2024-03-29T11:29:18.799 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadd1c33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadd1c33640 1 -- 172.21.15.67:0/527768068 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fadcc072480 con 0x7fadcc06f110 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.803+0000 7fadd1c33640 1 librados: init done 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.805+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadc0001930 con 0x7fadcc06f110 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.805+0000 7fadb3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.805+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fadc00037f0 con 0x7fadcc06f110 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.805+0000 7fadb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.805+0000 7fadb3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.805+0000 7fadb3fff640 10 monclient: dump: 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:18.800 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:18.801 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:18.801 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:18.801 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.805+0000 7fadc97fa640 4 set_mon_vals no callback set 2024-03-29T11:29:18.801 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.806+0000 7fadd1c33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:18.801 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.807+0000 7fadd1c33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:18.801 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.807+0000 7fadd1c33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:18.801 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.807+0000 7fadd1c33640 1 -- 172.21.15.67:0/527768068 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadac005080 con 0x7fadcc06f110 2024-03-29T11:29:18.802 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.810+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fadc0003950 con 0x7fadcc06f110 2024-03-29T11:29:18.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.811+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7fadc0084990 con 0x7fadcc06f110 2024-03-29T11:29:18.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.811+0000 7fadb3fff640 10 client.4563.objecter ms_dispatch 0x7fadcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:18.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.811+0000 7fadb3fff640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:18.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.811+0000 7fadb3fff640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:18.804 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.811+0000 7fadb3fff640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-29T11:29:18.805 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.813+0000 7fadcb7fe640 10 monclient: get_auth_request con 0x7fadb806dc70 auth_method 0 2024-03-29T11:29:18.805 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.814+0000 7fadb3fff640 10 client.4563.objecter ms_handle_connect 0x7fadb806dc70 2024-03-29T11:29:18.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.820+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fadc00500a0 con 0x7fadcc06f110 2024-03-29T11:29:18.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.820+0000 7fadb3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:18.811 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:18.820+0000 7fadb3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:19.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.175+0000 7fadd1c33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:19.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.175+0000 7fadd1c33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:19.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.175+0000 7fadd1c33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:19.166 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.175+0000 7fadd1c33640 1 -- 172.21.15.67:0/527768068 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadac005330 con 0x7fadcc06f110 2024-03-29T11:29:19.167 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.176+0000 7fadb3fff640 1 -- 172.21.15.67:0/527768068 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7fadc00521a0 con 0x7fadcc06f110 2024-03-29T11:29:19.167 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.176+0000 7fadb3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:19.167 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.176+0000 7fadb3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:19.167 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:29:19.168 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:19.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.180+0000 7fadd1c33640 10 librados: watch_flush enter 2024-03-29T11:29:19.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.180+0000 7fadd1c33640 10 librados: watch_flush exit 2024-03-29T11:29:19.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.180+0000 7fadd1c33640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-29T11:29:19.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.181+0000 7fadd1c33640 10 client.4563.objecter successfully canceled tick 2024-03-29T11:29:19.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.181+0000 7fadd1c33640 1 -- 172.21.15.67:0/527768068 >> v1:172.21.15.67:6816/404787197 conn(0x7fadb806dc70 legacy=0x7fadb8070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.181+0000 7fadd1c33640 10 monclient: shutdown 2024-03-29T11:29:19.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.181+0000 7fadd1c33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:19.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.181+0000 7fadd1c33640 1 -- 172.21.15.67:0/527768068 >> v1:172.21.15.67:6789/0 conn(0x7fadcc06f110 legacy=0x7fadcc071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.181+0000 7fadd1c33640 1 -- 172.21.15.67:0/527768068 shutdown_connections 2024-03-29T11:29:19.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.181+0000 7fadd1c33640 1 -- 172.21.15.67:0/527768068 >> 172.21.15.67:0/527768068 conn(0x7fadcc0b35e0 msgr2=0x7fadcc0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:19.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.181+0000 7fadd1c33640 1 -- 172.21.15.67:0/527768068 shutdown_connections 2024-03-29T11:29:19.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.181+0000 7fadd1c33640 1 -- 172.21.15.67:0/527768068 wait complete. 2024-03-29T11:29:19.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:19.181+0000 7fadd1c33640 1 librados: shutdown 2024-03-29T11:29:19.188 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:25.996 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:26.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.155+0000 7f148eff6640 10 monclient: get_monmap_and_config 2024-03-29T11:29:26.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.155+0000 7f148eff6640 10 monclient: build_initial_monmap 2024-03-29T11:29:26.147 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.155+0000 7f148eff6640 10 monclient: monmap: 2024-03-29T11:29:26.147 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:26.147 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:26.147 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:29:26.156592+0000 2024-03-29T11:29:26.147 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:29:26.156592+0000 2024-03-29T11:29:26.148 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:26.148 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:26.148 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:26.148 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:29:26.148 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:29:26.148 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:26.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.156+0000 7f148eff6640 1 Processor -- start 2024-03-29T11:29:26.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.156+0000 7f148eff6640 1 -- start start 2024-03-29T11:29:26.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.156+0000 7f148eff6640 10 monclient: init 2024-03-29T11:29:26.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.157+0000 7f148eff6640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:26.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.157+0000 7f148eff6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:26.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.157+0000 7f148eff6640 10 monclient(hunting): picked mon.noname-a con 0x7f14880b0860 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:26.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.157+0000 7f148eff6640 10 monclient(hunting): picked mon.noname-b con 0x7f148813ef30 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:26.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.157+0000 7f148eff6640 10 monclient(hunting): picked mon.noname-c con 0x7f14880af8e0 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:26.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.157+0000 7f148eff6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14880b0c40 con 0x7f14880b0860 2024-03-29T11:29:26.150 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.157+0000 7f148eff6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14880aed10 con 0x7f14880af8e0 2024-03-29T11:29:26.151 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.157+0000 7f148eff6640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14880afd50 con 0x7f148813ef30 2024-03-29T11:29:26.151 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.157+0000 7f148eff6640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:26.151 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.157+0000 7f148eff6640 10 monclient(hunting): authenticate will time out at 1426.453754s 2024-03-29T11:29:26.151 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.160+0000 7f1487fff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f148813ef30 0x7f14880af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:40653/0 (socket says 172.21.15.67:40653) 2024-03-29T11:29:26.151 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.160+0000 7f1487fff640 1 -- 172.21.15.67:0/2455913942 learned_addr learned my addr 172.21.15.67:0/2455913942 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:26.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.160+0000 7f14877fe640 1 -- 172.21.15.67:0/2455913942 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804936899 0 0) 0x7f14880afd50 con 0x7f148813ef30 2024-03-29T11:29:26.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.161+0000 7f14877fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.161+0000 7f14877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.161+0000 7f14877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.161+0000 7f14877fe640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:29:26.152 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.161+0000 7f14877fe640 1 -- 172.21.15.67:0/2455913942 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1470003520 con 0x7f148813ef30 2024-03-29T11:29:26.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.161+0000 7f14877fe640 1 -- 172.21.15.67:0/2455913942 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1083020516 0 0) 0x7f1470003520 con 0x7f148813ef30 2024-03-29T11:29:26.153 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.161+0000 7f14877fe640 1 -- 172.21.15.67:0/2455913942 >> v1:172.21.15.82:6789/0 conn(0x7f14880af8e0 legacy=0x7f1488149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:26.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f14877fe640 1 -- 172.21.15.67:0/2455913942 >> v1:172.21.15.67:6789/0 conn(0x7f14880b0860 legacy=0x7f148813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f14877fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:26.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f14877fe640 1 monclient: found mon.noname-b 2024-03-29T11:29:26.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f14877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:26.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f14877fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:29:26.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f14877fe640 1 -- 172.21.15.67:0/2455913942 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14880ab1b0 con 0x7f148813ef30 2024-03-29T11:29:26.156 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f14877fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:26.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f14877fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:26.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f14877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:26.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f148eff6640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:29:26.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f148eff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:26.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f14877fe640 1 -- 172.21.15.67:0/2455913942 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f1480001a10 con 0x7f148813ef30 2024-03-29T11:29:26.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f14877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:26.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f14877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:26.157 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.163+0000 7f14877fe640 10 monclient: dump: 2024-03-29T11:29:26.157 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:26.157 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:26.157 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:26.157 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:26.157 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:26.158 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:26.158 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:26.158 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:26.158 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:26.158 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:26.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.164+0000 7f148eff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:26.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.164+0000 7f14877fe640 1 -- 172.21.15.67:0/2455913942 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1480001ca0 con 0x7f148813ef30 2024-03-29T11:29:26.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.164+0000 7f14877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:26.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.164+0000 7f148eff6640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:26.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.164+0000 7f148eff6640 4 set_mon_vals no callback set 2024-03-29T11:29:26.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.165+0000 7f148eff6640 10 monclient: shutdown 2024-03-29T11:29:26.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.165+0000 7f148eff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:26.158 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.165+0000 7f148eff6640 1 -- 172.21.15.67:0/2455913942 >> v1:172.21.15.67:6790/0 conn(0x7f148813ef30 legacy=0x7f14880af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.159 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.167+0000 7f148eff6640 1 -- 172.21.15.67:0/2455913942 shutdown_connections 2024-03-29T11:29:26.159 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.167+0000 7f148eff6640 1 -- 172.21.15.67:0/2455913942 >> 172.21.15.67:0/2455913942 conn(0x7f148806b660 msgr2=0x7f148806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:26.159 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.167+0000 7f148eff6640 1 -- 172.21.15.67:0/2455913942 shutdown_connections 2024-03-29T11:29:26.160 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.167+0000 7f148eff6640 1 -- 172.21.15.67:0/2455913942 wait complete. 2024-03-29T11:29:26.160 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.168+0000 7f148eff6640 10 monclient: build_initial_monmap 2024-03-29T11:29:26.160 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.168+0000 7f148eff6640 10 monclient: monmap: 2024-03-29T11:29:26.160 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:26.160 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:26.161 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:29:26.161 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:29:26.161 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:26.161 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:26.161 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:26.161 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:29:26.161 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:29:26.161 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:26.161 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.169+0000 7f148eff6640 1 librados: starting msgr at 2024-03-29T11:29:26.161 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.169+0000 7f148eff6640 1 librados: starting objecter 2024-03-29T11:29:26.161 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.169+0000 7f148eff6640 1 Processor -- start 2024-03-29T11:29:26.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.169+0000 7f148eff6640 1 -- start start 2024-03-29T11:29:26.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.169+0000 7f148eff6640 1 librados: setting wanted keys 2024-03-29T11:29:26.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.169+0000 7f148eff6640 1 librados: calling monclient init 2024-03-29T11:29:26.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.169+0000 7f148eff6640 10 monclient: init 2024-03-29T11:29:26.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.170+0000 7f148eff6640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:26.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.170+0000 7f148eff6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:26.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.170+0000 7f148eff6640 10 monclient(hunting): picked mon.noname-c con 0x7f14880af8e0 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:26.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.170+0000 7f148eff6640 10 monclient(hunting): picked mon.noname-a con 0x7f14880b0860 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:26.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.170+0000 7f148eff6640 10 monclient(hunting): picked mon.noname-b con 0x7f148813ef30 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:26.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.170+0000 7f148eff6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14880b42b0 con 0x7f14880b0860 2024-03-29T11:29:26.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.170+0000 7f148eff6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148806d9d0 con 0x7f148813ef30 2024-03-29T11:29:26.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.170+0000 7f148eff6640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f148806dcb0 con 0x7f14880af8e0 2024-03-29T11:29:26.171 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.170+0000 7f148eff6640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:26.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.170+0000 7f148eff6640 10 monclient(hunting): authenticate will time out at 1426.466343s 2024-03-29T11:29:26.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.176+0000 7f148cd92640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f14880af8e0 0x7f14880b11d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:40731/0 (socket says 172.21.15.67:40731) 2024-03-29T11:29:26.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.176+0000 7f148cd92640 1 -- 172.21.15.67:0/4068519902 learned_addr learned my addr 172.21.15.67:0/4068519902 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:26.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.176+0000 7f14857fa640 10 client.?.objecter ms_handle_connect 0x7f14880af8e0 2024-03-29T11:29:26.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.176+0000 7f14857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:26.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.176+0000 7f14857fa640 10 client.?.objecter ms_handle_connect 0x7f14880b0860 2024-03-29T11:29:26.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.176+0000 7f14857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:26.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.176+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598208004 0 0) 0x7f148806dcb0 con 0x7f14880af8e0 2024-03-29T11:29:26.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.176+0000 7f14857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.176+0000 7f14857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.176+0000 7f14857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.176+0000 7f14857fa640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:29:26.172 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.176+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1474004eb0 con 0x7f14880af8e0 2024-03-29T11:29:26.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486936712 0 0) 0x7f14880b42b0 con 0x7f14880b0860 2024-03-29T11:29:26.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:29:26.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f148806dcb0 con 0x7f14880b0860 2024-03-29T11:29:26.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 879445929 0 0) 0x7f1474004eb0 con 0x7f14880af8e0 2024-03-29T11:29:26.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14880b42b0 con 0x7f14880af8e0 2024-03-29T11:29:26.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f1478001950 con 0x7f14880af8e0 2024-03-29T11:29:26.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:26.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:26.173 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 10 monclient(hunting): dump: 2024-03-29T11:29:26.173 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:26.173 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:26.174 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:26.174 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:26.174 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:26.174 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:26.174 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:26.174 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:26.174 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:26.174 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:26.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:26.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3607625133 0 0) 0x7f14880b42b0 con 0x7f14880af8e0 2024-03-29T11:29:26.174 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.177+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 >> v1:172.21.15.82:6789/0 conn(0x7f148813ef30 legacy=0x7f14881ef5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:26.175 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.179+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 >> v1:172.21.15.67:6789/0 conn(0x7f14880b0860 legacy=0x7f14881e4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.175 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.179+0000 7f14857fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:26.175 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.179+0000 7f14857fa640 1 monclient: found mon.c 2024-03-29T11:29:26.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.179+0000 7f14857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:26.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.179+0000 7f14857fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:29:26.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.179+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14880b30a0 con 0x7f14880af8e0 2024-03-29T11:29:26.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.179+0000 7f14857fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:26.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.179+0000 7f14857fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:26.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.179+0000 7f14857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:26.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.180+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1478001ab0 con 0x7f14880af8e0 2024-03-29T11:29:26.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.180+0000 7f14857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:26.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.180+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f14780038c0 con 0x7f14880af8e0 2024-03-29T11:29:26.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.180+0000 7f14857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:26.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.180+0000 7f14857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:26.176 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.180+0000 7f14867fc640 4 set_mon_vals no callback set 2024-03-29T11:29:26.177 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.180+0000 7f14857fa640 10 monclient: dump: 2024-03-29T11:29:26.177 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:26.177 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:26.177 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:26.177 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:26.177 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:26.177 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:26.177 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:26.177 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:26.177 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:26.177 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:26.177 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.181+0000 7f148eff6640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:29:26.177 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.181+0000 7f148eff6640 10 monclient: _renew_subs 2024-03-29T11:29:26.177 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.181+0000 7f148eff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:29:26.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.181+0000 7f148eff6640 1 -- 172.21.15.67:0/4068519902 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14880b35c0 con 0x7f14880af8e0 2024-03-29T11:29:26.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.181+0000 7f148eff6640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:26.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.181+0000 7f148eff6640 10 monclient: _renew_subs 2024-03-29T11:29:26.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.181+0000 7f148eff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:29:26.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.181+0000 7f148eff6640 1 -- 172.21.15.67:0/4068519902 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14880b1ac0 con 0x7f14880af8e0 2024-03-29T11:29:26.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.181+0000 7f148eff6640 1 librados: init done 2024-03-29T11:29:26.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.183+0000 7f148eff6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:26.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.183+0000 7f148eff6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:26.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.183+0000 7f148eff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:29:26.178 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.183+0000 7f148eff6640 1 -- 172.21.15.67:0/4068519902 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1488075f10 con 0x7f14880af8e0 2024-03-29T11:29:26.179 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.187+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f1478003a20 con 0x7f14880af8e0 2024-03-29T11:29:26.179 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.187+0000 7f1487fff640 10 monclient: get_auth_request con 0x7f147406da30 auth_method 0 2024-03-29T11:29:26.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.188+0000 7f14857fa640 10 client.4622.objecter ms_handle_connect 0x7f147406da30 2024-03-29T11:29:26.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.188+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1807135670 0 0) 0x7f1478084a00 con 0x7f14880af8e0 2024-03-29T11:29:26.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.188+0000 7f14857fa640 10 client.4622.objecter ms_dispatch 0x7f1488000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:26.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.188+0000 7f14857fa640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:26.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.188+0000 7f14857fa640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:26.180 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.188+0000 7f14857fa640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:29:26.183 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.192+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1478050110 con 0x7f14880af8e0 2024-03-29T11:29:26.183 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.192+0000 7f14857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:26.183 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.192+0000 7f14857fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:26.518 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.527+0000 7f148eff6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:26.519 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.527+0000 7f148eff6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:26.519 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.527+0000 7f148eff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:29:26.519 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.527+0000 7f148eff6640 1 -- 172.21.15.67:0/4068519902 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1488072330 con 0x7f14880af8e0 2024-03-29T11:29:26.519 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.528+0000 7f14857fa640 1 -- 172.21.15.67:0/4068519902 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4194689305) 0x7f1478052210 con 0x7f14880af8e0 2024-03-29T11:29:26.519 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.528+0000 7f14857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:26.519 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.528+0000 7f14857fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:26.521 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:29:26.522 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:19:29.197148+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:29.197148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:26.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.534+0000 7f148eff6640 10 librados: watch_flush enter 2024-03-29T11:29:26.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.534+0000 7f148eff6640 10 librados: watch_flush exit 2024-03-29T11:29:26.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.534+0000 7f148eff6640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:29:26.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.535+0000 7f148eff6640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:29:26.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.535+0000 7f148eff6640 1 -- 172.21.15.67:0/4068519902 >> v1:172.21.15.67:6816/404787197 conn(0x7f147406da30 legacy=0x7f147406fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.535+0000 7f148eff6640 10 monclient: shutdown 2024-03-29T11:29:26.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.535+0000 7f148eff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:26.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.535+0000 7f148eff6640 1 -- 172.21.15.67:0/4068519902 >> v1:172.21.15.67:6790/0 conn(0x7f14880af8e0 legacy=0x7f14880b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.535+0000 7f148eff6640 1 -- 172.21.15.67:0/4068519902 shutdown_connections 2024-03-29T11:29:26.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.535+0000 7f148eff6640 1 -- 172.21.15.67:0/4068519902 >> 172.21.15.67:0/4068519902 conn(0x7f148806b660 msgr2=0x7f148806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:26.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.535+0000 7f148eff6640 1 -- 172.21.15.67:0/4068519902 shutdown_connections 2024-03-29T11:29:26.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.535+0000 7f148eff6640 1 -- 172.21.15.67:0/4068519902 wait complete. 2024-03-29T11:29:26.527 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:26.536+0000 7f148eff6640 1 librados: shutdown 2024-03-29T11:29:26.544 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:33.352 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:33.497 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.505+0000 7f0605974640 10 monclient: get_monmap_and_config 2024-03-29T11:29:33.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.505+0000 7f0605974640 10 monclient: build_initial_monmap 2024-03-29T11:29:33.498 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.505+0000 7f0605974640 10 monclient: monmap: 2024-03-29T11:29:33.498 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:33.498 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:33.498 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:29:33.507360+0000 2024-03-29T11:29:33.498 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:29:33.507360+0000 2024-03-29T11:29:33.498 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:33.498 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:33.498 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:33.498 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:29:33.498 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:29:33.499 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:33.501 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.506+0000 7f0605974640 1 Processor -- start 2024-03-29T11:29:33.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.506+0000 7f0605974640 1 -- start start 2024-03-29T11:29:33.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.506+0000 7f0605974640 10 monclient: init 2024-03-29T11:29:33.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.507+0000 7f0605974640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:33.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.507+0000 7f0605974640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:33.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.507+0000 7f0605974640 10 monclient(hunting): picked mon.noname-b con 0x7f06000b2790 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:33.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.507+0000 7f0605974640 10 monclient(hunting): picked mon.noname-c con 0x7f0600143c80 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:33.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.507+0000 7f0605974640 10 monclient(hunting): picked mon.noname-a con 0x7f0600146e30 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:33.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.507+0000 7f0605974640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0600146290 con 0x7f0600146e30 2024-03-29T11:29:33.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.507+0000 7f0605974640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06000b3970 con 0x7f0600143c80 2024-03-29T11:29:33.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.507+0000 7f0605974640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f060006ee80 con 0x7f06000b2790 2024-03-29T11:29:33.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.507+0000 7f0605974640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:33.502 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.507+0000 7f0605974640 10 monclient(hunting): authenticate will time out at 1433.804561s 2024-03-29T11:29:33.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.511+0000 7f05fffff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f0600143c80 0x7f0600146720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:39752/0 (socket says 172.21.15.67:39752) 2024-03-29T11:29:33.503 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.511+0000 7f05fffff640 1 -- 172.21.15.67:0/2375890279 learned_addr learned my addr 172.21.15.67:0/2375890279 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:33.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 1 -- 172.21.15.67:0/2375890279 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98789167 0 0) 0x7f06000b3970 con 0x7f0600143c80 2024-03-29T11:29:33.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:33.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:33.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:33.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:29:33.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 1 -- 172.21.15.67:0/2375890279 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05e8003520 con 0x7f0600143c80 2024-03-29T11:29:33.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 1 -- 172.21.15.67:0/2375890279 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1688019462 0 0) 0x7f05e8003520 con 0x7f0600143c80 2024-03-29T11:29:33.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 1 -- 172.21.15.67:0/2375890279 >> v1:172.21.15.67:6790/0 conn(0x7f06000b2790 legacy=0x7f0600143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:33.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 1 -- 172.21.15.67:0/2375890279 >> v1:172.21.15.67:6789/0 conn(0x7f0600146e30 legacy=0x7f0600151bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:33.506 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 1 monclient: found mon.noname-c 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 1 -- 172.21.15.67:0/2375890279 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060006f160 con 0x7f0600143c80 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 1 -- 172.21.15.67:0/2375890279 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f05f8001a10 con 0x7f0600143c80 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f05ff7fe640 10 monclient: dump: 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:33.507 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:33.508 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:33.508 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:33.508 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:33.508 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:33.508 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:33.508 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:33.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f0605974640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:29:33.508 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.512+0000 7f0605974640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:33.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.514+0000 7f05ff7fe640 1 -- 172.21.15.67:0/2375890279 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05f8001ca0 con 0x7f0600143c80 2024-03-29T11:29:33.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.514+0000 7f05ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:33.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.514+0000 7f05ff7fe640 1 -- 172.21.15.67:0/2375890279 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f05f8002030 con 0x7f0600143c80 2024-03-29T11:29:33.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.514+0000 7f05ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:33.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.514+0000 7f05ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:33.509 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.514+0000 7f05ff7fe640 10 monclient: dump: 2024-03-29T11:29:33.509 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:33.509 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:33.509 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:33.509 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:33.509 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:33.509 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:33.509 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:33.509 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:33.510 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:33.510 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:33.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.514+0000 7f0605974640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:33.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.514+0000 7f0605974640 4 set_mon_vals no callback set 2024-03-29T11:29:33.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.515+0000 7f0605974640 10 monclient: shutdown 2024-03-29T11:29:33.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.515+0000 7f0605974640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:33.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.515+0000 7f0605974640 1 -- 172.21.15.67:0/2375890279 >> v1:172.21.15.82:6789/0 conn(0x7f0600143c80 legacy=0x7f0600146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:33.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.517+0000 7f0605974640 1 -- 172.21.15.67:0/2375890279 shutdown_connections 2024-03-29T11:29:33.510 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.517+0000 7f0605974640 1 -- 172.21.15.67:0/2375890279 >> 172.21.15.67:0/2375890279 conn(0x7f060006d070 msgr2=0x7f0600066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:33.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.518+0000 7f0605974640 1 -- 172.21.15.67:0/2375890279 shutdown_connections 2024-03-29T11:29:33.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.519+0000 7f0605974640 1 -- 172.21.15.67:0/2375890279 wait complete. 2024-03-29T11:29:33.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.519+0000 7f0605974640 10 monclient: build_initial_monmap 2024-03-29T11:29:33.513 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.519+0000 7f0605974640 10 monclient: monmap: 2024-03-29T11:29:33.513 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:33.513 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:33.513 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:29:33.513 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:29:33.513 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:33.513 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:33.513 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:33.513 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:29:33.514 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:29:33.514 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:33.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.520+0000 7f0605974640 1 librados: starting msgr at 2024-03-29T11:29:33.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.520+0000 7f0605974640 1 librados: starting objecter 2024-03-29T11:29:33.514 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.520+0000 7f0605974640 1 Processor -- start 2024-03-29T11:29:33.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.520+0000 7f0605974640 1 -- start start 2024-03-29T11:29:33.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.520+0000 7f0605974640 1 librados: setting wanted keys 2024-03-29T11:29:33.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.520+0000 7f0605974640 1 librados: calling monclient init 2024-03-29T11:29:33.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.520+0000 7f0605974640 10 monclient: init 2024-03-29T11:29:33.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.521+0000 7f0605974640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:33.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.521+0000 7f0605974640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:33.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.521+0000 7f0605974640 10 monclient(hunting): picked mon.noname-a con 0x7f06000b2790 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:33.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.521+0000 7f0605974640 10 monclient(hunting): picked mon.noname-c con 0x7f0600143c80 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:33.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.521+0000 7f0605974640 10 monclient(hunting): picked mon.noname-b con 0x7f0600146e30 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:33.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.521+0000 7f0605974640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06001e4a90 con 0x7f06000b2790 2024-03-29T11:29:33.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.521+0000 7f0605974640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0600148950 con 0x7f0600146e30 2024-03-29T11:29:33.515 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.521+0000 7f0605974640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06001e3b80 con 0x7f0600143c80 2024-03-29T11:29:33.516 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.521+0000 7f0605974640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:33.516 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.521+0000 7f0605974640 10 monclient(hunting): authenticate will time out at 1433.818658s 2024-03-29T11:29:33.516 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.524+0000 7f0604972640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f06000b2790 0x7f06001e4380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38537/0 (socket says 172.21.15.67:38537) 2024-03-29T11:29:33.516 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.524+0000 7f0604972640 1 -- 172.21.15.67:0/2449536992 learned_addr learned my addr 172.21.15.67:0/2449536992 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:33.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.524+0000 7f05fd7fa640 10 client.?.objecter ms_handle_connect 0x7f06000b2790 2024-03-29T11:29:33.517 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.525+0000 7f05fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:33.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.528+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893905151 0 0) 0x7f06001e4a90 con 0x7f06000b2790 2024-03-29T11:29:33.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.528+0000 7f05fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:33.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.528+0000 7f05fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:33.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.528+0000 7f05fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:33.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.529+0000 7f05fd7fa640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:29:33.523 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.529+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05ec004eb0 con 0x7f06000b2790 2024-03-29T11:29:33.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.529+0000 7f05fd7fa640 10 client.?.objecter ms_handle_connect 0x7f0600143c80 2024-03-29T11:29:33.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.529+0000 7f05fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:33.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.529+0000 7f05fd7fa640 10 client.?.objecter ms_handle_connect 0x7f0600146e30 2024-03-29T11:29:33.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.529+0000 7f05fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:33.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.529+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609966405 0 0) 0x7f06001e3b80 con 0x7f0600143c80 2024-03-29T11:29:33.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.529+0000 7f05fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:33.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.529+0000 7f05fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:33.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.529+0000 7f05fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:33.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.529+0000 7f05fd7fa640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:29:33.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.529+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06001e4a90 con 0x7f0600143c80 2024-03-29T11:29:33.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.529+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369670229 0 0) 0x7f0600148950 con 0x7f0600146e30 2024-03-29T11:29:33.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.530+0000 7f05fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:33.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.530+0000 7f05fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:33.524 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.530+0000 7f05fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:33.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.530+0000 7f05fd7fa640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:29:33.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.530+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06001e3b80 con 0x7f0600146e30 2024-03-29T11:29:33.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.530+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 693928371 0 0) 0x7f05ec004eb0 con 0x7f06000b2790 2024-03-29T11:29:33.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.530+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0600148950 con 0x7f06000b2790 2024-03-29T11:29:33.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.530+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f05f0001950 con 0x7f06000b2790 2024-03-29T11:29:33.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.530+0000 7f05fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:33.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.530+0000 7f05fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:33.525 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.530+0000 7f05fd7fa640 10 monclient(hunting): dump: 2024-03-29T11:29:33.525 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:33.525 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:33.525 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:33.525 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:33.525 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:33.525 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:33.526 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:33.526 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:33.526 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:33.526 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:33.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.530+0000 7f05fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:33.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.531+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3151819196 0 0) 0x7f06001e3b80 con 0x7f0600146e30 2024-03-29T11:29:33.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.531+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05ec004eb0 con 0x7f0600146e30 2024-03-29T11:29:33.526 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.531+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f05f40015c0 con 0x7f0600146e30 2024-03-29T11:29:33.529 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.531+0000 7f05fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:33.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.531+0000 7f05fd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:33.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.531+0000 7f05fd7fa640 10 monclient(hunting): dump: 2024-03-29T11:29:33.530 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:33.530 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:33.530 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:33.530 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:33.530 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:33.530 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:33.530 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:33.530 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:33.530 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:33.530 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:33.530 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.532+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4222512407 0 0) 0x7f05ec004eb0 con 0x7f0600146e30 2024-03-29T11:29:33.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.532+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 >> v1:172.21.15.67:6790/0 conn(0x7f0600143c80 legacy=0x7f06001dacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:33.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.532+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 >> v1:172.21.15.67:6789/0 conn(0x7f06000b2790 legacy=0x7f06001e4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:33.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.532+0000 7f05fd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:33.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.532+0000 7f05fd7fa640 1 monclient: found mon.b 2024-03-29T11:29:33.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.532+0000 7f05fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:33.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.532+0000 7f05fd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:33.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.532+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06001e3e60 con 0x7f0600146e30 2024-03-29T11:29:33.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.532+0000 7f05fd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:33.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.532+0000 7f05fd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:33.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.532+0000 7f05fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:33.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f0605974640 5 monclient: authenticate success, global_id 4528 2024-03-29T11:29:33.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f0605974640 10 monclient: _renew_subs 2024-03-29T11:29:33.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f0605974640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:33.531 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f0605974640 1 -- 172.21.15.67:0/2449536992 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060014a5a0 con 0x7f0600146e30 2024-03-29T11:29:33.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f0605974640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:33.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f0605974640 10 monclient: _renew_subs 2024-03-29T11:29:33.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f0605974640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:33.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f0605974640 1 -- 172.21.15.67:0/2449536992 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06001db5e0 con 0x7f0600146e30 2024-03-29T11:29:33.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f0605974640 1 librados: init done 2024-03-29T11:29:33.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05f4001ad0 con 0x7f0600146e30 2024-03-29T11:29:33.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f05fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:33.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f05f40035f0 con 0x7f0600146e30 2024-03-29T11:29:33.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f05fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:33.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f05fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:33.532 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.533+0000 7f05fd7fa640 10 monclient: dump: 2024-03-29T11:29:33.532 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:33.532 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:33.532 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:33.533 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:33.533 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:33.533 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:33.533 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:33.533 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:33.533 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:33.533 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:33.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.534+0000 7f05feffd640 4 set_mon_vals no callback set 2024-03-29T11:29:33.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.536+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f05f40195c0 con 0x7f0600146e30 2024-03-29T11:29:33.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.537+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3315246757 0 0) 0x7f05f40833b0 con 0x7f0600146e30 2024-03-29T11:29:33.533 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.537+0000 7f05fd7fa640 10 client.4528.objecter ms_dispatch 0x7f0600000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:33.534 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.537+0000 7f0604972640 10 monclient: get_auth_request con 0x7f05ec06dfa0 auth_method 0 2024-03-29T11:29:33.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.538+0000 7f05fd7fa640 3 client.4528.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:33.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.538+0000 7f05fd7fa640 3 client.4528.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:33.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.538+0000 7f05fd7fa640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-29T11:29:33.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.538+0000 7f05fd7fa640 10 client.4528.objecter ms_handle_connect 0x7f05ec06dfa0 2024-03-29T11:29:33.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.538+0000 7f0605974640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:33.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.538+0000 7f0605974640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:33.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.538+0000 7f0605974640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:33.535 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.539+0000 7f0605974640 1 -- 172.21.15.67:0/2449536992 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e0005080 con 0x7f0600146e30 2024-03-29T11:29:33.542 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.549+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f05f404ec90 con 0x7f0600146e30 2024-03-29T11:29:33.542 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.549+0000 7f05fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:33.542 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.549+0000 7f05fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:33.876 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.883+0000 7f0605974640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:33.876 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.883+0000 7f0605974640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:33.876 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.883+0000 7f0605974640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:33.876 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.883+0000 7f0605974640 1 -- 172.21.15.67:0/2449536992 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05e0005330 con 0x7f0600146e30 2024-03-29T11:29:33.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.888+0000 7f05fd7fa640 1 -- 172.21.15.67:0/2449536992 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1631012267) 0x7f05f4050bc0 con 0x7f0600146e30 2024-03-29T11:29:33.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.888+0000 7f05fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:33.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.888+0000 7f05fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:33.881 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:29:33.881 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":11,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:29:31.896474+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:31.896474+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:33.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.892+0000 7f0605974640 10 librados: watch_flush enter 2024-03-29T11:29:33.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.895+0000 7f0605974640 10 librados: watch_flush exit 2024-03-29T11:29:33.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.895+0000 7f0605974640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-29T11:29:33.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.895+0000 7f0605974640 10 client.4528.objecter successfully canceled tick 2024-03-29T11:29:33.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.896+0000 7f0605974640 1 -- 172.21.15.67:0/2449536992 >> v1:172.21.15.67:6816/404787197 conn(0x7f05ec06dfa0 legacy=0x7f05ec070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:33.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.896+0000 7f0605974640 10 monclient: shutdown 2024-03-29T11:29:33.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.896+0000 7f0605974640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:33.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.896+0000 7f0605974640 1 -- 172.21.15.67:0/2449536992 >> v1:172.21.15.82:6789/0 conn(0x7f0600146e30 legacy=0x7f06001ef3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:33.888 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.896+0000 7f0605974640 1 -- 172.21.15.67:0/2449536992 shutdown_connections 2024-03-29T11:29:33.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.896+0000 7f0605974640 1 -- 172.21.15.67:0/2449536992 >> 172.21.15.67:0/2449536992 conn(0x7f060006d070 msgr2=0x7f0600066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:33.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.896+0000 7f0605974640 1 -- 172.21.15.67:0/2449536992 shutdown_connections 2024-03-29T11:29:33.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.896+0000 7f0605974640 1 -- 172.21.15.67:0/2449536992 wait complete. 2024-03-29T11:29:33.889 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:33.896+0000 7f0605974640 1 librados: shutdown 2024-03-29T11:29:33.911 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:40.718 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:40.871 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.879+0000 7fa623460640 10 monclient: get_monmap_and_config 2024-03-29T11:29:40.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.879+0000 7fa623460640 10 monclient: build_initial_monmap 2024-03-29T11:29:40.872 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.879+0000 7fa623460640 10 monclient: monmap: 2024-03-29T11:29:40.872 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:40.872 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:40.872 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:29:40.880905+0000 2024-03-29T11:29:40.872 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:29:40.880905+0000 2024-03-29T11:29:40.872 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:40.872 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:40.872 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:40.872 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:29:40.872 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:29:40.872 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:40.873 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.879+0000 7fa623460640 1 Processor -- start 2024-03-29T11:29:40.873 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.880+0000 7fa623460640 1 -- start start 2024-03-29T11:29:40.873 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.880+0000 7fa623460640 10 monclient: init 2024-03-29T11:29:40.873 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.881+0000 7fa623460640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:40.873 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.881+0000 7fa623460640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:40.873 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.881+0000 7fa623460640 10 monclient(hunting): picked mon.noname-a con 0x7fa61c1422b0 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:40.873 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.881+0000 7fa623460640 10 monclient(hunting): picked mon.noname-c con 0x7fa61c142690 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:40.874 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.881+0000 7fa623460640 10 monclient(hunting): picked mon.noname-b con 0x7fa61c0b17d0 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:40.874 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.881+0000 7fa623460640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61c142c80 con 0x7fa61c1422b0 2024-03-29T11:29:40.874 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.881+0000 7fa623460640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61c0aae40 con 0x7fa61c142690 2024-03-29T11:29:40.874 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.881+0000 7fa623460640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61c13fbd0 con 0x7fa61c0b17d0 2024-03-29T11:29:40.874 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.881+0000 7fa623460640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:40.874 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.881+0000 7fa623460640 10 monclient(hunting): authenticate will time out at 1441.178136s 2024-03-29T11:29:40.877 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.884+0000 7fa6219fd640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fa61c0b17d0 0x7fa61c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:45876/0 (socket says 172.21.15.67:45876) 2024-03-29T11:29:40.877 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.884+0000 7fa6219fd640 1 -- 172.21.15.67:0/3188277279 learned_addr learned my addr 172.21.15.67:0/3188277279 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:40.877 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.885+0000 7fa613fff640 1 -- 172.21.15.67:0/3188277279 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877873015 0 0) 0x7fa61c13fbd0 con 0x7fa61c0b17d0 2024-03-29T11:29:40.877 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.885+0000 7fa613fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:40.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.885+0000 7fa613fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:40.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.885+0000 7fa613fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:40.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.885+0000 7fa613fff640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:29:40.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.885+0000 7fa613fff640 1 -- 172.21.15.67:0/3188277279 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa608003520 con 0x7fa61c0b17d0 2024-03-29T11:29:40.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.885+0000 7fa613fff640 1 -- 172.21.15.67:0/3188277279 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1168299629 0 0) 0x7fa608003520 con 0x7fa61c0b17d0 2024-03-29T11:29:40.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.885+0000 7fa613fff640 1 -- 172.21.15.67:0/3188277279 >> v1:172.21.15.82:6789/0 conn(0x7fa61c142690 legacy=0x7fa61c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:40.878 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.885+0000 7fa613fff640 1 -- 172.21.15.67:0/3188277279 >> v1:172.21.15.67:6789/0 conn(0x7fa61c1422b0 legacy=0x7fa61c13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:40.880 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa613fff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:40.880 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa613fff640 1 monclient: found mon.noname-b 2024-03-29T11:29:40.880 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa613fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:40.880 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa613fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:29:40.880 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa613fff640 1 -- 172.21.15.67:0/3188277279 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61c151940 con 0x7fa61c0b17d0 2024-03-29T11:29:40.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa613fff640 10 monclient: _finish_auth 0 2024-03-29T11:29:40.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa613fff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:40.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa613fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:40.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa623460640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:29:40.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa623460640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:40.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa613fff640 1 -- 172.21.15.67:0/3188277279 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa614001a10 con 0x7fa61c0b17d0 2024-03-29T11:29:40.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa613fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:40.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa613fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:40.881 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa613fff640 10 monclient: dump: 2024-03-29T11:29:40.881 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:40.881 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:40.881 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:40.881 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:40.881 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:40.882 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:40.882 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:40.882 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:40.882 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:40.882 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:40.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa623460640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:40.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa613fff640 1 -- 172.21.15.67:0/3188277279 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa614001c70 con 0x7fa61c0b17d0 2024-03-29T11:29:40.882 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa613fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:40.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa623460640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:40.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.888+0000 7fa623460640 4 set_mon_vals no callback set 2024-03-29T11:29:40.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.889+0000 7fa623460640 10 monclient: shutdown 2024-03-29T11:29:40.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.889+0000 7fa623460640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:40.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.889+0000 7fa623460640 1 -- 172.21.15.67:0/3188277279 >> v1:172.21.15.67:6790/0 conn(0x7fa61c0b17d0 legacy=0x7fa61c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:40.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.889+0000 7fa623460640 1 -- 172.21.15.67:0/3188277279 shutdown_connections 2024-03-29T11:29:40.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.889+0000 7fa623460640 1 -- 172.21.15.67:0/3188277279 >> 172.21.15.67:0/3188277279 conn(0x7fa61c066a00 msgr2=0x7fa61c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:40.883 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.891+0000 7fa623460640 1 -- 172.21.15.67:0/3188277279 shutdown_connections 2024-03-29T11:29:40.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.891+0000 7fa623460640 1 -- 172.21.15.67:0/3188277279 wait complete. 2024-03-29T11:29:40.884 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.892+0000 7fa623460640 10 monclient: build_initial_monmap 2024-03-29T11:29:40.885 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.892+0000 7fa623460640 10 monclient: monmap: 2024-03-29T11:29:40.885 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:40.885 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:40.885 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:29:40.885 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:29:40.885 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:40.885 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:40.885 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:40.885 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:29:40.885 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:29:40.885 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:40.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.893+0000 7fa623460640 1 librados: starting msgr at 2024-03-29T11:29:40.886 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.893+0000 7fa623460640 1 librados: starting objecter 2024-03-29T11:29:40.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.893+0000 7fa623460640 1 Processor -- start 2024-03-29T11:29:40.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.893+0000 7fa623460640 1 -- start start 2024-03-29T11:29:40.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.893+0000 7fa623460640 1 librados: setting wanted keys 2024-03-29T11:29:40.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.893+0000 7fa623460640 1 librados: calling monclient init 2024-03-29T11:29:40.887 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.893+0000 7fa623460640 10 monclient: init 2024-03-29T11:29:40.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.894+0000 7fa623460640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:40.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.894+0000 7fa623460640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:40.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.894+0000 7fa623460640 10 monclient(hunting): picked mon.noname-b con 0x7fa61c0b17d0 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:40.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.894+0000 7fa623460640 10 monclient(hunting): picked mon.noname-a con 0x7fa61c1422b0 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:40.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.894+0000 7fa623460640 10 monclient(hunting): picked mon.noname-c con 0x7fa61c142690 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:40.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.894+0000 7fa623460640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61c071e00 con 0x7fa61c1422b0 2024-03-29T11:29:40.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.894+0000 7fa623460640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61c072050 con 0x7fa61c0b17d0 2024-03-29T11:29:40.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.894+0000 7fa623460640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61c070420 con 0x7fa61c142690 2024-03-29T11:29:40.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.894+0000 7fa623460640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:40.891 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.894+0000 7fa623460640 10 monclient(hunting): authenticate will time out at 1441.191032s 2024-03-29T11:29:40.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.896+0000 7fa6219fd640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fa61c142690 0x7fa61c1e6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:49064/0 (socket says 172.21.15.67:49064) 2024-03-29T11:29:40.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.896+0000 7fa6219fd640 1 -- 172.21.15.67:0/97301040 learned_addr learned my addr 172.21.15.67:0/97301040 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:40.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.897+0000 7fa611ffb640 10 client.?.objecter ms_handle_connect 0x7fa61c142690 2024-03-29T11:29:40.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.897+0000 7fa611ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:40.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.897+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808984988 0 0) 0x7fa61c070420 con 0x7fa61c142690 2024-03-29T11:29:40.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.897+0000 7fa611ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:40.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.897+0000 7fa611ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:40.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.897+0000 7fa611ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:40.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.897+0000 7fa611ffb640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:29:40.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.897+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa600004eb0 con 0x7fa61c142690 2024-03-29T11:29:40.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.898+0000 7fa611ffb640 10 client.?.objecter ms_handle_connect 0x7fa61c0b17d0 2024-03-29T11:29:40.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.898+0000 7fa611ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:40.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.898+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450203463 0 0) 0x7fa61c072050 con 0x7fa61c0b17d0 2024-03-29T11:29:40.892 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.898+0000 7fa611ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:40.893 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.898+0000 7fa611ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:40.893 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.898+0000 7fa611ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:40.893 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.898+0000 7fa611ffb640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:29:40.893 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.898+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa61c070420 con 0x7fa61c0b17d0 2024-03-29T11:29:40.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.900+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1097321556 0 0) 0x7fa61c070420 con 0x7fa61c0b17d0 2024-03-29T11:29:40.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.900+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa61c072050 con 0x7fa61c0b17d0 2024-03-29T11:29:40.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.900+0000 7fa611ffb640 10 client.?.objecter ms_handle_connect 0x7fa61c1422b0 2024-03-29T11:29:40.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.900+0000 7fa611ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:40.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.900+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa60c001830 con 0x7fa61c0b17d0 2024-03-29T11:29:40.897 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:40.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:40.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 10 monclient(hunting): dump: 2024-03-29T11:29:40.898 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:40.898 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:40.898 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:40.898 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:40.898 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:40.898 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:40.898 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:40.898 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:40.898 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:40.898 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:40.898 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:40.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 244573240 0 0) 0x7fa61c072050 con 0x7fa61c0b17d0 2024-03-29T11:29:40.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 >> v1:172.21.15.67:6790/0 conn(0x7fa61c142690 legacy=0x7fa61c1e6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:40.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 >> v1:172.21.15.67:6789/0 conn(0x7fa61c1422b0 legacy=0x7fa61c131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:40.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:40.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 1 monclient: found mon.b 2024-03-29T11:29:40.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:40.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:40.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61c06e5a0 con 0x7fa61c0b17d0 2024-03-29T11:29:40.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:40.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:40.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.901+0000 7fa611ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:40.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa623460640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:29:40.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa623460640 10 monclient: _renew_subs 2024-03-29T11:29:40.899 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa623460640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:40.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa623460640 1 -- 172.21.15.67:0/97301040 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61c070ce0 con 0x7fa61c0b17d0 2024-03-29T11:29:40.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa623460640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:40.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa623460640 10 monclient: _renew_subs 2024-03-29T11:29:40.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa623460640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:40.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa623460640 1 -- 172.21.15.67:0/97301040 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa61c0b6c90 con 0x7fa61c0b17d0 2024-03-29T11:29:40.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa623460640 1 librados: init done 2024-03-29T11:29:40.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa60c001990 con 0x7fa61c0b17d0 2024-03-29T11:29:40.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa611ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:40.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa612ffd640 4 set_mon_vals no callback set 2024-03-29T11:29:40.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa60c003790 con 0x7fa61c0b17d0 2024-03-29T11:29:40.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa611ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:40.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa611ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:40.900 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.902+0000 7fa611ffb640 10 monclient: dump: 2024-03-29T11:29:40.901 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:40.901 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:40.901 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:40.901 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:40.901 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:40.901 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:40.901 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:40.901 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:40.901 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:40.901 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:40.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.903+0000 7fa623460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:40.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.903+0000 7fa623460640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:40.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.903+0000 7fa623460640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:40.901 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.903+0000 7fa623460640 1 -- 172.21.15.67:0/97301040 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5fc005080 con 0x7fa61c0b17d0 2024-03-29T11:29:40.902 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.905+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fa60c019760 con 0x7fa61c0b17d0 2024-03-29T11:29:40.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.906+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3315246757 0 0) 0x7fa60c083780 con 0x7fa61c0b17d0 2024-03-29T11:29:40.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.906+0000 7fa611ffb640 10 client.4531.objecter ms_dispatch 0x7fa61c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:40.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.906+0000 7fa611ffb640 3 client.4531.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:40.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.906+0000 7fa611ffb640 3 client.4531.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:40.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.906+0000 7fa611ffb640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-29T11:29:40.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.907+0000 7fa6209fb640 10 monclient: get_auth_request con 0x7fa60006dd00 auth_method 0 2024-03-29T11:29:40.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.907+0000 7fa611ffb640 10 client.4531.objecter ms_handle_connect 0x7fa60006dd00 2024-03-29T11:29:40.903 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.911+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa60c04ee90 con 0x7fa61c0b17d0 2024-03-29T11:29:40.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.911+0000 7fa611ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:40.904 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:40.911+0000 7fa611ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:41.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.250+0000 7fa623460640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:41.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.250+0000 7fa623460640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:41.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.250+0000 7fa623460640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:29:41.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.250+0000 7fa623460640 1 -- 172.21.15.67:0/97301040 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5fc005330 con 0x7fa61c0b17d0 2024-03-29T11:29:41.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.252+0000 7fa611ffb640 1 -- 172.21.15.67:0/97301040 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1631012267) 0x7fa60c050f90 con 0x7fa61c0b17d0 2024-03-29T11:29:41.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.252+0000 7fa611ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:41.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.252+0000 7fa611ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:41.246 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:29:41.246 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":11,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:29:31.896474+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:31.896474+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:41.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.255+0000 7fa623460640 10 librados: watch_flush enter 2024-03-29T11:29:41.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.255+0000 7fa623460640 10 librados: watch_flush exit 2024-03-29T11:29:41.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.255+0000 7fa623460640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-29T11:29:41.248 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.255+0000 7fa623460640 10 client.4531.objecter successfully canceled tick 2024-03-29T11:29:41.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.255+0000 7fa623460640 1 -- 172.21.15.67:0/97301040 >> v1:172.21.15.67:6816/404787197 conn(0x7fa60006dd00 legacy=0x7fa6000701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.255+0000 7fa623460640 10 monclient: shutdown 2024-03-29T11:29:41.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.255+0000 7fa623460640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:41.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.255+0000 7fa623460640 1 -- 172.21.15.67:0/97301040 >> v1:172.21.15.82:6789/0 conn(0x7fa61c0b17d0 legacy=0x7fa61c06de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.256+0000 7fa623460640 1 -- 172.21.15.67:0/97301040 shutdown_connections 2024-03-29T11:29:41.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.256+0000 7fa623460640 1 -- 172.21.15.67:0/97301040 >> 172.21.15.67:0/97301040 conn(0x7fa61c066a00 msgr2=0x7fa61c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:41.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.256+0000 7fa623460640 1 -- 172.21.15.67:0/97301040 shutdown_connections 2024-03-29T11:29:41.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.256+0000 7fa623460640 1 -- 172.21.15.67:0/97301040 wait complete. 2024-03-29T11:29:41.249 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:41.257+0000 7fa623460640 1 librados: shutdown 2024-03-29T11:29:41.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:48.073 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:48.218 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.226+0000 7fa7b3d6c640 10 monclient: get_monmap_and_config 2024-03-29T11:29:48.218 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.226+0000 7fa7b3d6c640 10 monclient: build_initial_monmap 2024-03-29T11:29:48.218 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.226+0000 7fa7b3d6c640 10 monclient: monmap: 2024-03-29T11:29:48.219 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:48.219 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:48.219 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:29:48.227665+0000 2024-03-29T11:29:48.219 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:29:48.227665+0000 2024-03-29T11:29:48.219 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:48.219 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:48.219 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:48.219 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:29:48.219 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:29:48.219 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:48.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.227+0000 7fa7b3d6c640 1 Processor -- start 2024-03-29T11:29:48.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.228+0000 7fa7b3d6c640 1 -- start start 2024-03-29T11:29:48.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.228+0000 7fa7b3d6c640 10 monclient: init 2024-03-29T11:29:48.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.228+0000 7fa7b3d6c640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:48.221 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.228+0000 7fa7b3d6c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:48.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.228+0000 7fa7b3d6c640 10 monclient(hunting): picked mon.noname-b con 0x7fa7ac1422b0 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:48.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.228+0000 7fa7b3d6c640 10 monclient(hunting): picked mon.noname-a con 0x7fa7ac142690 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:48.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.228+0000 7fa7b3d6c640 10 monclient(hunting): picked mon.noname-c con 0x7fa7ac0b17d0 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:48.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.228+0000 7fa7b3d6c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac142c80 con 0x7fa7ac142690 2024-03-29T11:29:48.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.228+0000 7fa7b3d6c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac0aae40 con 0x7fa7ac0b17d0 2024-03-29T11:29:48.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.228+0000 7fa7b3d6c640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac13fbd0 con 0x7fa7ac1422b0 2024-03-29T11:29:48.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.228+0000 7fa7b3d6c640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:48.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.228+0000 7fa7b3d6c640 10 monclient(hunting): authenticate will time out at 1448.524842s 2024-03-29T11:29:48.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b2309640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fa7ac0b17d0 0x7fa7ac0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:54934/0 (socket says 172.21.15.67:54934) 2024-03-29T11:29:48.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b2309640 1 -- 172.21.15.67:0/3468277468 learned_addr learned my addr 172.21.15.67:0/3468277468 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:48.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b1b08640 1 -- 172.21.15.67:0/3468277468 >> v1:172.21.15.67:6790/0 conn(0x7fa7ac1422b0 legacy=0x7fa7ac13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-29T11:29:48.222 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b1b08640 1 -- 172.21.15.67:0/3468277468 >> v1:172.21.15.67:6790/0 conn(0x7fa7ac1422b0 legacy=0x7fa7ac13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T11:29:48.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b1b08640 1 --1- 172.21.15.67:0/3468277468 >> v1:172.21.15.67:6790/0 conn(0x7fa7ac1422b0 0x7fa7ac13f4c0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-29T11:29:48.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b0b06640 1 -- 172.21.15.67:0/3468277468 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306453210 0 0) 0x7fa7ac0aae40 con 0x7fa7ac0b17d0 2024-03-29T11:29:48.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b0b06640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b0b06640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b0b06640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b0b06640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:29:48.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b0b06640 1 -- 172.21.15.67:0/3468277468 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa798003520 con 0x7fa7ac0b17d0 2024-03-29T11:29:48.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b0b06640 1 -- 172.21.15.67:0/3468277468 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448269304 0 0) 0x7fa7ac142c80 con 0x7fa7ac142690 2024-03-29T11:29:48.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b0b06640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b0b06640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b0b06640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b0b06640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:29:48.223 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.229+0000 7fa7b0b06640 1 -- 172.21.15.67:0/3468277468 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7ac0aae40 con 0x7fa7ac142690 2024-03-29T11:29:48.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 1 -- 172.21.15.67:0/3468277468 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1639883332 0 0) 0x7fa798003520 con 0x7fa7ac0b17d0 2024-03-29T11:29:48.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 1 -- 172.21.15.67:0/3468277468 >> v1:172.21.15.67:6790/0 conn(0x7fa7ac1422b0 legacy=0x7fa7ac13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:48.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 1 -- 172.21.15.67:0/3468277468 >> v1:172.21.15.67:6789/0 conn(0x7fa7ac142690 legacy=0x7fa7ac0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 10 monclient: _finish_hunting 0 2024-03-29T11:29:48.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 1 monclient: found mon.noname-c 2024-03-29T11:29:48.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:48.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T11:29:48.225 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 1 -- 172.21.15.67:0/3468277468 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ac151940 con 0x7fa7ac0b17d0 2024-03-29T11:29:48.226 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 10 monclient: _finish_auth 0 2024-03-29T11:29:48.226 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 10 monclient: _check_auth_tickets 2024-03-29T11:29:48.226 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:48.226 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 1 -- 172.21.15.67:0/3468277468 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa7a4001a10 con 0x7fa7ac0b17d0 2024-03-29T11:29:48.226 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.226 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:48.226 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b0b06640 10 monclient: dump: 2024-03-29T11:29:48.226 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:48.226 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:48.227 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:48.227 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:48.227 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.227 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:48.227 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:48.227 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:48.227 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:48.227 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:48.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b3d6c640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:29:48.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.230+0000 7fa7b3d6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:48.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.231+0000 7fa7b0b06640 1 -- 172.21.15.67:0/3468277468 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7a4001b70 con 0x7fa7ac0b17d0 2024-03-29T11:29:48.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.231+0000 7fa7b0b06640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:48.227 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.231+0000 7fa7b0b06640 1 -- 172.21.15.67:0/3468277468 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa7a4001f40 con 0x7fa7ac0b17d0 2024-03-29T11:29:48.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.231+0000 7fa7b0b06640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.231+0000 7fa7b0b06640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:48.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.231+0000 7fa7b0b06640 10 monclient: dump: 2024-03-29T11:29:48.228 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:48.228 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:48.228 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:48.228 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:48.228 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.228 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:48.228 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:48.228 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:48.228 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:48.228 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:48.228 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.231+0000 7fa7b3d6c640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:48.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.231+0000 7fa7b3d6c640 4 set_mon_vals no callback set 2024-03-29T11:29:48.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.232+0000 7fa7b3d6c640 10 monclient: shutdown 2024-03-29T11:29:48.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.232+0000 7fa7b3d6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:48.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.232+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/3468277468 >> v1:172.21.15.82:6789/0 conn(0x7fa7ac0b17d0 legacy=0x7fa7ac0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.232+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/3468277468 shutdown_connections 2024-03-29T11:29:48.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.232+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/3468277468 >> 172.21.15.67:0/3468277468 conn(0x7fa7ac066a00 msgr2=0x7fa7ac066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:48.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.232+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/3468277468 shutdown_connections 2024-03-29T11:29:48.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.232+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/3468277468 wait complete. 2024-03-29T11:29:48.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.233+0000 7fa7b3d6c640 10 monclient: build_initial_monmap 2024-03-29T11:29:48.229 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.233+0000 7fa7b3d6c640 10 monclient: monmap: 2024-03-29T11:29:48.229 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:48.229 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:48.229 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:29:48.229 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:29:48.230 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:48.230 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:48.230 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:48.230 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:29:48.230 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:29:48.230 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:48.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.234+0000 7fa7b3d6c640 1 librados: starting msgr at 2024-03-29T11:29:48.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.234+0000 7fa7b3d6c640 1 librados: starting objecter 2024-03-29T11:29:48.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.234+0000 7fa7b3d6c640 1 Processor -- start 2024-03-29T11:29:48.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.234+0000 7fa7b3d6c640 1 -- start start 2024-03-29T11:29:48.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.234+0000 7fa7b3d6c640 1 librados: setting wanted keys 2024-03-29T11:29:48.233 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.234+0000 7fa7b3d6c640 1 librados: calling monclient init 2024-03-29T11:29:48.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.234+0000 7fa7b3d6c640 10 monclient: init 2024-03-29T11:29:48.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.235+0000 7fa7b3d6c640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:48.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.235+0000 7fa7b3d6c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:48.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.235+0000 7fa7b3d6c640 10 monclient(hunting): picked mon.noname-c con 0x7fa7ac0b17d0 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:48.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.235+0000 7fa7b3d6c640 10 monclient(hunting): picked mon.noname-b con 0x7fa7ac1422b0 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:48.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.235+0000 7fa7b3d6c640 10 monclient(hunting): picked mon.noname-a con 0x7fa7ac142690 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:48.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.235+0000 7fa7b3d6c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac06e460 con 0x7fa7ac142690 2024-03-29T11:29:48.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.235+0000 7fa7b3d6c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac12f680 con 0x7fa7ac1422b0 2024-03-29T11:29:48.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.235+0000 7fa7b3d6c640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac06e6d0 con 0x7fa7ac0b17d0 2024-03-29T11:29:48.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.235+0000 7fa7b1b08640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fa7ac0b17d0 0x7fa7ac0b5830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:58766/0 (socket says 172.21.15.67:58766) 2024-03-29T11:29:48.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.235+0000 7fa7b1b08640 1 -- 172.21.15.67:0/2964980831 learned_addr learned my addr 172.21.15.67:0/2964980831 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:48.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.235+0000 7fa7b3d6c640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:48.234 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.235+0000 7fa7b3d6c640 10 monclient(hunting): authenticate will time out at 1448.531505s 2024-03-29T11:29:48.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.235+0000 7fa7a27fc640 10 client.?.objecter ms_handle_connect 0x7fa7ac0b17d0 2024-03-29T11:29:48.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.235+0000 7fa7a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:48.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 client.?.objecter ms_handle_connect 0x7fa7ac142690 2024-03-29T11:29:48.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:48.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997791353 0 0) 0x7fa7ac06e6d0 con 0x7fa7ac0b17d0 2024-03-29T11:29:48.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:29:48.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa790004eb0 con 0x7fa7ac0b17d0 2024-03-29T11:29:48.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542131256 0 0) 0x7fa7ac06e460 con 0x7fa7ac142690 2024-03-29T11:29:48.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.235 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:29:48.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7ac06e6d0 con 0x7fa7ac142690 2024-03-29T11:29:48.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 client.?.objecter ms_handle_connect 0x7fa7ac1422b0 2024-03-29T11:29:48.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:48.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1882340409 0 0) 0x7fa7ac06e6d0 con 0x7fa7ac142690 2024-03-29T11:29:48.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7ac06e460 con 0x7fa7ac142690 2024-03-29T11:29:48.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa7a4001e80 con 0x7fa7ac142690 2024-03-29T11:29:48.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:48.236 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 monclient(hunting): dump: 2024-03-29T11:29:48.236 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:48.236 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:48.236 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:48.236 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:48.237 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.237 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:48.237 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:48.237 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:48.237 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:48.237 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:48.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.236+0000 7fa7a27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:48.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2903698779 0 0) 0x7fa790004eb0 con 0x7fa7ac0b17d0 2024-03-29T11:29:48.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7ac06e6d0 con 0x7fa7ac0b17d0 2024-03-29T11:29:48.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa7a80040a0 con 0x7fa7ac0b17d0 2024-03-29T11:29:48.237 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.238 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:48.238 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 10 monclient(hunting): dump: 2024-03-29T11:29:48.238 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:48.238 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:48.238 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:48.238 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:48.238 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.238 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:48.238 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:48.239 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:48.239 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:48.239 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:48.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285100810 0 0) 0x7fa7ac12f680 con 0x7fa7ac1422b0 2024-03-29T11:29:48.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:29:48.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa790004eb0 con 0x7fa7ac1422b0 2024-03-29T11:29:48.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2628405687 0 0) 0x7fa7ac06e460 con 0x7fa7ac142690 2024-03-29T11:29:48.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 >> v1:172.21.15.67:6790/0 conn(0x7fa7ac0b17d0 legacy=0x7fa7ac0b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 >> v1:172.21.15.82:6789/0 conn(0x7fa7ac1422b0 legacy=0x7fa7ac12e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.239 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:48.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 1 monclient: found mon.a 2024-03-29T11:29:48.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:48.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:48.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ac06e9b0 con 0x7fa7ac142690 2024-03-29T11:29:48.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:48.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:48.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:48.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7b3d6c640 5 monclient: authenticate success, global_id 4578 2024-03-29T11:29:48.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7b3d6c640 10 monclient: _renew_subs 2024-03-29T11:29:48.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7b3d6c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:48.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/2964980831 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ac06ec90 con 0x7fa7ac142690 2024-03-29T11:29:48.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7b3d6c640 10 client.4578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:48.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7b3d6c640 10 monclient: _renew_subs 2024-03-29T11:29:48.240 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7b3d6c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:48.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/2964980831 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ac1502c0 con 0x7fa7ac142690 2024-03-29T11:29:48.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.237+0000 7fa7b3d6c640 1 librados: init done 2024-03-29T11:29:48.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.238+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7a4001fe0 con 0x7fa7ac142690 2024-03-29T11:29:48.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.238+0000 7fa7a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:48.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.238+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa7a4004e10 con 0x7fa7ac142690 2024-03-29T11:29:48.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.238+0000 7fa7a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.238+0000 7fa7a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:48.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.238+0000 7fa7a3fff640 4 set_mon_vals no callback set 2024-03-29T11:29:48.241 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.238+0000 7fa7a27fc640 10 monclient: dump: 2024-03-29T11:29:48.241 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:48.241 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:48.241 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:48.241 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:48.241 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.242 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:48.242 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:48.242 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:48.242 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:48.242 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:48.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.238+0000 7fa7b3d6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:48.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.239+0000 7fa7b3d6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:48.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.239+0000 7fa7b3d6c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:48.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.239+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/2964980831 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa794005080 con 0x7fa7ac142690 2024-03-29T11:29:48.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.240+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fa7a401ade0 con 0x7fa7ac142690 2024-03-29T11:29:48.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.241+0000 7fa7b1b08640 10 monclient: get_auth_request con 0x7fa79006dfe0 auth_method 0 2024-03-29T11:29:48.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.241+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3315246757 0 0) 0x7fa7a4084da0 con 0x7fa7ac142690 2024-03-29T11:29:48.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.241+0000 7fa7a27fc640 10 client.4578.objecter ms_dispatch 0x7fa7ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:48.242 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.241+0000 7fa7a27fc640 3 client.4578.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:48.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.241+0000 7fa7a27fc640 3 client.4578.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:48.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.241+0000 7fa7a27fc640 20 client.4578.objecter dump_active .. 0 homeless 2024-03-29T11:29:48.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.241+0000 7fa7a27fc640 10 client.4578.objecter ms_handle_connect 0x7fa79006dfe0 2024-03-29T11:29:48.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.248+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7a40504b0 con 0x7fa7ac142690 2024-03-29T11:29:48.243 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.248+0000 7fa7a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:48.244 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.248+0000 7fa7a27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:48.580 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.588+0000 7fa7b3d6c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:48.580 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.589+0000 7fa7b3d6c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:48.581 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.589+0000 7fa7b3d6c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:48.581 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.589+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/2964980831 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa794005330 con 0x7fa7ac142690 2024-03-29T11:29:48.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.591+0000 7fa7a27fc640 1 -- 172.21.15.67:0/2964980831 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1631012267) 0x7fa7a40525b0 con 0x7fa7ac142690 2024-03-29T11:29:48.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.591+0000 7fa7a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:48.583 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.591+0000 7fa7a27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:48.583 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:29:48.583 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":11,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:29:31.896474+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:31.896474+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:48.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.596+0000 7fa7b3d6c640 10 librados: watch_flush enter 2024-03-29T11:29:48.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.596+0000 7fa7b3d6c640 10 librados: watch_flush exit 2024-03-29T11:29:48.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.596+0000 7fa7b3d6c640 20 client.4578.objecter shutdown clearing up homeless session... 2024-03-29T11:29:48.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.596+0000 7fa7b3d6c640 10 client.4578.objecter successfully canceled tick 2024-03-29T11:29:48.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.596+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/2964980831 >> v1:172.21.15.67:6816/404787197 conn(0x7fa79006dfe0 legacy=0x7fa7900704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.596+0000 7fa7b3d6c640 10 monclient: shutdown 2024-03-29T11:29:48.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.596+0000 7fa7b3d6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:48.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.596+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/2964980831 >> v1:172.21.15.67:6789/0 conn(0x7fa7ac142690 legacy=0x7fa7ac12ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.597+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/2964980831 shutdown_connections 2024-03-29T11:29:48.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.597+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/2964980831 >> 172.21.15.67:0/2964980831 conn(0x7fa7ac066a00 msgr2=0x7fa7ac132600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:48.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.598+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/2964980831 shutdown_connections 2024-03-29T11:29:48.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.598+0000 7fa7b3d6c640 1 -- 172.21.15.67:0/2964980831 wait complete. 2024-03-29T11:29:48.590 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:48.598+0000 7fa7b3d6c640 1 librados: shutdown 2024-03-29T11:29:48.610 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:55.418 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:55.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.580+0000 7f037adc4640 10 monclient: get_monmap_and_config 2024-03-29T11:29:55.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.580+0000 7f037adc4640 10 monclient: build_initial_monmap 2024-03-29T11:29:55.573 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.580+0000 7f037adc4640 10 monclient: monmap: 2024-03-29T11:29:55.573 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:55.573 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:55.573 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:29:55.580857+0000 2024-03-29T11:29:55.573 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:29:55.580857+0000 2024-03-29T11:29:55.573 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:55.573 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:55.574 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:55.574 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:29:55.574 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:29:55.574 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:55.575 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.583+0000 7f037adc4640 1 Processor -- start 2024-03-29T11:29:55.576 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.584+0000 7f037adc4640 1 -- start start 2024-03-29T11:29:55.576 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.584+0000 7f037adc4640 10 monclient: init 2024-03-29T11:29:55.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.585+0000 7f037adc4640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:55.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.585+0000 7f037adc4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:55.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.585+0000 7f037adc4640 10 monclient(hunting): picked mon.noname-b con 0x7f03740ae250 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:55.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.585+0000 7f037adc4640 10 monclient(hunting): picked mon.noname-a con 0x7f037413f970 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:55.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.585+0000 7f037adc4640 10 monclient(hunting): picked mon.noname-c con 0x7f0374142800 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:55.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.585+0000 7f037adc4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03740afe00 con 0x7f037413f970 2024-03-29T11:29:55.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.585+0000 7f037adc4640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0374142c10 con 0x7f0374142800 2024-03-29T11:29:55.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.585+0000 7f037adc4640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03740afb20 con 0x7f03740ae250 2024-03-29T11:29:55.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.585+0000 7f037adc4640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:55.578 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.585+0000 7f037adc4640 10 monclient(hunting): authenticate will time out at 1455.881578s 2024-03-29T11:29:55.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.590+0000 7f0378b60640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f03740ae250 0x7f037413f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:50931/0 (socket says 172.21.15.67:50931) 2024-03-29T11:29:55.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.590+0000 7f0378b60640 1 -- 172.21.15.67:0/1748597757 learned_addr learned my addr 172.21.15.67:0/1748597757 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:55.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 1 -- 172.21.15.67:0/1748597757 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684582892 0 0) 0x7f03740afb20 con 0x7f03740ae250 2024-03-29T11:29:55.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:55.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:55.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:55.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:29:55.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 1 -- 172.21.15.67:0/1748597757 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f035c003520 con 0x7f03740ae250 2024-03-29T11:29:55.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 1 -- 172.21.15.67:0/1748597757 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 207805872 0 0) 0x7f035c003520 con 0x7f03740ae250 2024-03-29T11:29:55.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 1 -- 172.21.15.67:0/1748597757 >> v1:172.21.15.82:6789/0 conn(0x7f0374142800 legacy=0x7f037414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:55.584 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 1 -- 172.21.15.67:0/1748597757 >> v1:172.21.15.67:6789/0 conn(0x7f037413f970 legacy=0x7f03741420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:55.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:55.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 1 monclient: found mon.noname-b 2024-03-29T11:29:55.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:55.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:29:55.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 1 -- 172.21.15.67:0/1748597757 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03740aab80 con 0x7f03740ae250 2024-03-29T11:29:55.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:55.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:55.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:55.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 1 -- 172.21.15.67:0/1748597757 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0364001690 con 0x7f03740ae250 2024-03-29T11:29:55.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:55.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:55.585 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.591+0000 7f03737fe640 10 monclient: dump: 2024-03-29T11:29:55.585 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:55.586 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:55.586 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:55.586 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:55.586 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:55.586 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:55.586 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:55.586 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:55.586 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:55.586 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:55.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.592+0000 7f03737fe640 1 -- 172.21.15.67:0/1748597757 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0364001860 con 0x7f03740ae250 2024-03-29T11:29:55.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.592+0000 7f03737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:55.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.592+0000 7f03737fe640 1 -- 172.21.15.67:0/1748597757 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0364001cb0 con 0x7f03740ae250 2024-03-29T11:29:55.586 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.592+0000 7f03737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:55.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.592+0000 7f03737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:55.587 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.592+0000 7f03737fe640 10 monclient: dump: 2024-03-29T11:29:55.587 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:55.587 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:55.587 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:55.587 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:55.587 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:55.587 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:55.587 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:55.587 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:55.587 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:55.587 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:55.588 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.592+0000 7f037adc4640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:29:55.588 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.592+0000 7f037adc4640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:55.588 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.592+0000 7f037adc4640 4 set_mon_vals no callback set 2024-03-29T11:29:55.588 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.593+0000 7f037adc4640 10 monclient: shutdown 2024-03-29T11:29:55.588 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.593+0000 7f037adc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:55.588 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.593+0000 7f037adc4640 1 -- 172.21.15.67:0/1748597757 >> v1:172.21.15.67:6790/0 conn(0x7f03740ae250 legacy=0x7f037413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:55.588 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.594+0000 7f037adc4640 1 -- 172.21.15.67:0/1748597757 shutdown_connections 2024-03-29T11:29:55.588 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.594+0000 7f037adc4640 1 -- 172.21.15.67:0/1748597757 >> 172.21.15.67:0/1748597757 conn(0x7f03740b35e0 msgr2=0x7f03740b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:55.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.597+0000 7f037adc4640 1 -- 172.21.15.67:0/1748597757 shutdown_connections 2024-03-29T11:29:55.589 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.598+0000 7f037adc4640 1 -- 172.21.15.67:0/1748597757 wait complete. 2024-03-29T11:29:55.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.598+0000 7f037adc4640 10 monclient: build_initial_monmap 2024-03-29T11:29:55.591 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.598+0000 7f037adc4640 10 monclient: monmap: 2024-03-29T11:29:55.591 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:29:55.591 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:55.591 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:29:55.592 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:29:55.592 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:55.592 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:55.592 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:29:55.592 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:29:55.592 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:29:55.592 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:55.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.599+0000 7f037adc4640 1 librados: starting msgr at 2024-03-29T11:29:55.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.599+0000 7f037adc4640 1 librados: starting objecter 2024-03-29T11:29:55.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.599+0000 7f037adc4640 1 Processor -- start 2024-03-29T11:29:55.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.599+0000 7f037adc4640 1 -- start start 2024-03-29T11:29:55.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.599+0000 7f037adc4640 1 librados: setting wanted keys 2024-03-29T11:29:55.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.599+0000 7f037adc4640 1 librados: calling monclient init 2024-03-29T11:29:55.592 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.599+0000 7f037adc4640 10 monclient: init 2024-03-29T11:29:55.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.600+0000 7f037adc4640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:55.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.600+0000 7f037adc4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:55.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.600+0000 7f037adc4640 10 monclient(hunting): picked mon.noname-b con 0x7f03740ae250 addr v1:172.21.15.82:6789/0 2024-03-29T11:29:55.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.600+0000 7f037adc4640 10 monclient(hunting): picked mon.noname-a con 0x7f037413f970 addr v1:172.21.15.67:6789/0 2024-03-29T11:29:55.594 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.600+0000 7f037adc4640 10 monclient(hunting): picked mon.noname-c con 0x7f0374142800 addr v1:172.21.15.67:6790/0 2024-03-29T11:29:55.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.600+0000 7f037adc4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03741423e0 con 0x7f037413f970 2024-03-29T11:29:55.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.600+0000 7f037adc4640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03740ac3a0 con 0x7f03740ae250 2024-03-29T11:29:55.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.600+0000 7f037adc4640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03740abf10 con 0x7f0374142800 2024-03-29T11:29:55.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.600+0000 7f037adc4640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:55.595 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.600+0000 7f037adc4640 10 monclient(hunting): authenticate will time out at 1455.896625s 2024-03-29T11:29:55.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.604+0000 7f0378b60640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f03740ae250 0x7f03741339c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:38347/0 (socket says 172.21.15.67:38347) 2024-03-29T11:29:55.597 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.604+0000 7f0378b60640 1 -- 172.21.15.67:0/549409807 learned_addr learned my addr 172.21.15.67:0/549409807 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:29:55.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 10 client.?.objecter ms_handle_connect 0x7f037413f970 2024-03-29T11:29:55.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:55.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 10 client.?.objecter ms_handle_connect 0x7f03740ae250 2024-03-29T11:29:55.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:55.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397046901 0 0) 0x7f03741423e0 con 0x7f037413f970 2024-03-29T11:29:55.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:55.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:55.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:55.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:29:55.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0360004eb0 con 0x7f037413f970 2024-03-29T11:29:55.600 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658989162 0 0) 0x7f03740ac3a0 con 0x7f03740ae250 2024-03-29T11:29:55.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:55.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:55.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:55.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:29:55.601 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.607+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03741423e0 con 0x7f03740ae250 2024-03-29T11:29:55.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 10 client.?.objecter ms_handle_connect 0x7f0374142800 2024-03-29T11:29:55.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:55.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4188293915 0 0) 0x7f0360004eb0 con 0x7f037413f970 2024-03-29T11:29:55.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03740ac3a0 con 0x7f037413f970 2024-03-29T11:29:55.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0368001870 con 0x7f037413f970 2024-03-29T11:29:55.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:55.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:55.602 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 10 monclient(hunting): dump: 2024-03-29T11:29:55.602 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:55.603 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:55.603 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:55.603 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:55.603 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:55.603 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:55.603 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:55.603 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:55.603 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:55.603 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:55.603 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:55.603 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042990258 0 0) 0x7f03740abf10 con 0x7f0374142800 2024-03-29T11:29:55.603 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:55.603 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:55.603 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.608+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0360004eb0 con 0x7f0374142800 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.609+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4146540307 0 0) 0x7f03741423e0 con 0x7f03740ae250 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.609+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03740abf10 con 0x7f03740ae250 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.609+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0364001b70 con 0x7f03740ae250 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.609+0000 7f03717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.609+0000 7f03717fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.609+0000 7f03717fa640 10 monclient(hunting): dump: 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:55.604 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:55.605 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:55.605 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:55.605 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:55.607 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 <== mon.0 v1:172.21.15.67:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 221141079 0 0) 0x7f03740ac3a0 con 0x7f037413f970 2024-03-29T11:29:55.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 >> v1:172.21.15.67:6790/0 conn(0x7f0374142800 legacy=0x7f0374140700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:55.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 >> v1:172.21.15.82:6789/0 conn(0x7f03740ae250 legacy=0x7f03741339c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:55.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:55.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 1 monclient: found mon.a 2024-03-29T11:29:55.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:55.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:55.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03740ac0a0 con 0x7f037413f970 2024-03-29T11:29:55.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:55.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:55.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:55.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f037adc4640 5 monclient: authenticate success, global_id 4581 2024-03-29T11:29:55.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f037adc4640 10 monclient: _renew_subs 2024-03-29T11:29:55.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f037adc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:55.608 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f037adc4640 1 -- 172.21.15.67:0/549409807 --> v1:172.21.15.67:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03741340d0 con 0x7f037413f970 2024-03-29T11:29:55.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f037adc4640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:55.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f037adc4640 10 monclient: _renew_subs 2024-03-29T11:29:55.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f037adc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:55.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f037adc4640 1 -- 172.21.15.67:0/549409807 --> v1:172.21.15.67:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0374139cc0 con 0x7f037413f970 2024-03-29T11:29:55.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f037adc4640 1 librados: init done 2024-03-29T11:29:55.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 <== mon.0 v1:172.21.15.67:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03680035f0 con 0x7f037413f970 2024-03-29T11:29:55.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:55.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 <== mon.0 v1:172.21.15.67:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f0368001d80 con 0x7f037413f970 2024-03-29T11:29:55.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:55.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:55.609 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f03717fa640 10 monclient: dump: 2024-03-29T11:29:55.609 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:29:55.609 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:29:55.609 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:55.610 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:29:55.610 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:55.610 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:29:55.610 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:29:55.610 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:29:55.610 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:29:55.610 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:29:55.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.611+0000 7f0372ffd640 4 set_mon_vals no callback set 2024-03-29T11:29:55.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.614+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 <== mon.0 v1:172.21.15.67:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f0368003900 con 0x7f037413f970 2024-03-29T11:29:55.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.615+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 <== mon.0 v1:172.21.15.67:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3315246757 0 0) 0x7f03680837c0 con 0x7f037413f970 2024-03-29T11:29:55.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.615+0000 7f03717fa640 10 client.4581.objecter ms_dispatch 0x7f0374000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:55.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.615+0000 7f03717fa640 3 client.4581.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:55.610 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.615+0000 7f03717fa640 3 client.4581.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:55.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.615+0000 7f03717fa640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-29T11:29:55.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.617+0000 7f0378b60640 10 monclient: get_auth_request con 0x7f036006def0 auth_method 0 2024-03-29T11:29:55.611 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.618+0000 7f03717fa640 10 client.4581.objecter ms_handle_connect 0x7f036006def0 2024-03-29T11:29:55.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.621+0000 7f037adc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:55.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.621+0000 7f037adc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:55.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.621+0000 7f037adc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:55.614 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.621+0000 7f037adc4640 1 -- 172.21.15.67:0/549409807 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0354005080 con 0x7f037413f970 2024-03-29T11:29:55.622 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.629+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 <== mon.0 v1:172.21.15.67:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f036804eed0 con 0x7f037413f970 2024-03-29T11:29:55.622 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.630+0000 7f03717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:55.622 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.630+0000 7f03717fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:55.963 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.971+0000 7f037adc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:55.963 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.972+0000 7f037adc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:55.964 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.972+0000 7f037adc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.67:6789/0 2024-03-29T11:29:55.964 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.972+0000 7f037adc4640 1 -- 172.21.15.67:0/549409807 --> v1:172.21.15.67:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0354005330 con 0x7f037413f970 2024-03-29T11:29:55.965 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.973+0000 7f03717fa640 1 -- 172.21.15.67:0/549409807 <== mon.0 v1:172.21.15.67:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1631012267) 0x7f0368050fd0 con 0x7f037413f970 2024-03-29T11:29:55.965 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.973+0000 7f03717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:55.965 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.973+0000 7f03717fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:55.965 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:29:55.965 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":11,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:29:31.896474+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:31.896474+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.976+0000 7f037adc4640 10 librados: watch_flush enter 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.976+0000 7f037adc4640 10 librados: watch_flush exit 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.976+0000 7f037adc4640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.976+0000 7f037adc4640 10 client.4581.objecter successfully canceled tick 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.976+0000 7f037adc4640 1 -- 172.21.15.67:0/549409807 >> v1:172.21.15.67:6816/404787197 conn(0x7f036006def0 legacy=0x7f03600703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.977+0000 7f037adc4640 10 monclient: shutdown 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.977+0000 7f037adc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.977+0000 7f037adc4640 1 -- 172.21.15.67:0/549409807 >> v1:172.21.15.67:6789/0 conn(0x7f037413f970 legacy=0x7f037413fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.977+0000 7f037adc4640 1 -- 172.21.15.67:0/549409807 shutdown_connections 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.977+0000 7f037adc4640 1 -- 172.21.15.67:0/549409807 >> 172.21.15.67:0/549409807 conn(0x7f03740b35e0 msgr2=0x7f03740b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.977+0000 7f037adc4640 1 -- 172.21.15.67:0/549409807 shutdown_connections 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.977+0000 7f037adc4640 1 -- 172.21.15.67:0/549409807 wait complete. 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:29:55.977+0000 7f037adc4640 1 librados: shutdown 2024-03-29T11:29:55.983 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:02.790 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:02.941 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.948+0000 7f4ee15eb640 10 monclient: get_monmap_and_config 2024-03-29T11:30:02.941 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.948+0000 7f4ee15eb640 10 monclient: build_initial_monmap 2024-03-29T11:30:02.941 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.948+0000 7f4ee15eb640 10 monclient: monmap: 2024-03-29T11:30:02.941 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:30:02.941 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:02.941 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:30:02.949141+0000 2024-03-29T11:30:02.941 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:30:02.949141+0000 2024-03-29T11:30:02.941 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:02.942 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:02.942 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:30:02.942 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:30:02.942 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:30:02.942 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:02.943 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.950+0000 7f4ee15eb640 1 Processor -- start 2024-03-29T11:30:02.943 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.950+0000 7f4ee15eb640 1 -- start start 2024-03-29T11:30:02.943 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.950+0000 7f4ee15eb640 10 monclient: init 2024-03-29T11:30:02.943 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.952+0000 7f4ee15eb640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:02.944 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.952+0000 7f4ee15eb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:02.948 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.955+0000 7f4ee15eb640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed4034850 addr v1:172.21.15.67:6789/0 2024-03-29T11:30:02.948 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.955+0000 7f4edbfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4ed4034850 0x7f4ed40c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54963/0 (socket says 172.21.15.67:54963) 2024-03-29T11:30:02.948 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.955+0000 7f4edbfff640 1 -- 172.21.15.67:0/2949204243 learned_addr learned my addr 172.21.15.67:0/2949204243 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:30:02.948 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.956+0000 7f4ee15eb640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed4006410 addr v1:172.21.15.82:6789/0 2024-03-29T11:30:02.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.956+0000 7f4ee15eb640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed40327d0 addr v1:172.21.15.67:6790/0 2024-03-29T11:30:02.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.956+0000 7f4ee15eb640 1 -- 172.21.15.67:0/2949204243 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed402c820 con 0x7f4ed4034850 2024-03-29T11:30:02.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.956+0000 7f4ee15eb640 1 -- 172.21.15.67:0/2949204243 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed402a120 con 0x7f4ed4006410 2024-03-29T11:30:02.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.956+0000 7f4ee15eb640 1 -- 172.21.15.67:0/2949204243 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed40c75b0 con 0x7f4ed40327d0 2024-03-29T11:30:02.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.956+0000 7f4ee15eb640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:02.949 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.956+0000 7f4ee15eb640 10 monclient(hunting): authenticate will time out at 1463.252371s 2024-03-29T11:30:02.958 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.958+0000 7f4edaffd640 1 -- 172.21.15.67:0/2949204243 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010337934 0 0) 0x7f4ed40c75b0 con 0x7f4ed40327d0 2024-03-29T11:30:02.958 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.958+0000 7f4edaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:02.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.958+0000 7f4edaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:02.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.958+0000 7f4edaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:02.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.958+0000 7f4edaffd640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:30:02.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.958+0000 7f4edaffd640 1 -- 172.21.15.67:0/2949204243 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ecc003440 con 0x7f4ed40327d0 2024-03-29T11:30:02.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 1 -- 172.21.15.67:0/2949204243 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3655094065 0 0) 0x7f4ecc003440 con 0x7f4ed40327d0 2024-03-29T11:30:02.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 1 -- 172.21.15.67:0/2949204243 >> v1:172.21.15.82:6789/0 conn(0x7f4ed4006410 legacy=0x7f4ed40320c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:02.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 1 -- 172.21.15.67:0/2949204243 >> v1:172.21.15.67:6789/0 conn(0x7f4ed4034850 legacy=0x7f4ed40c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:02.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:02.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 1 monclient: found mon.noname-b 2024-03-29T11:30:02.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:02.959 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.67:6790/0 2024-03-29T11:30:02.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 1 -- 172.21.15.67:0/2949204243 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed4033110 con 0x7f4ed40327d0 2024-03-29T11:30:02.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:02.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:02.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:02.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 1 -- 172.21.15.67:0/2949204243 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f4edc04f730 con 0x7f4ed40327d0 2024-03-29T11:30:02.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:02.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:02.960 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 10 monclient: dump: 2024-03-29T11:30:02.960 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:02.960 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:02.960 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:02.961 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:02.961 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:02.961 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:02.961 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:02.961 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:02.961 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:02.961 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:02.961 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 1 -- 172.21.15.67:0/2949204243 <== mon.2 v1:172.21.15.67:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4edc0008d0 con 0x7f4ed40327d0 2024-03-29T11:30:02.961 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:02.961 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 1 -- 172.21.15.67:0/2949204243 <== mon.2 v1:172.21.15.67:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f4edc0758b0 con 0x7f4ed40327d0 2024-03-29T11:30:02.961 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:02.961 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:02.961 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.959+0000 7f4edaffd640 10 monclient: dump: 2024-03-29T11:30:02.962 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:02.962 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:02.962 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:02.962 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:02.962 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:02.962 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:02.962 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:02.962 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:02.962 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:02.962 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:02.962 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.960+0000 7f4ee15eb640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:30:02.962 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.960+0000 7f4ee15eb640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:02.963 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.960+0000 7f4ee15eb640 4 set_mon_vals no callback set 2024-03-29T11:30:02.963 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.961+0000 7f4ee15eb640 10 monclient: shutdown 2024-03-29T11:30:02.963 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.961+0000 7f4ee15eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:02.963 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.961+0000 7f4ee15eb640 1 -- 172.21.15.67:0/2949204243 >> v1:172.21.15.67:6790/0 conn(0x7f4ed40327d0 legacy=0x7f4ed40cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:02.963 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.961+0000 7f4ee15eb640 1 -- 172.21.15.67:0/2949204243 shutdown_connections 2024-03-29T11:30:02.963 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.961+0000 7f4ee15eb640 1 -- 172.21.15.67:0/2949204243 >> 172.21.15.67:0/2949204243 conn(0x7f4ed40058f0 msgr2=0x7f4ed4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:02.963 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.962+0000 7f4ee15eb640 1 -- 172.21.15.67:0/2949204243 shutdown_connections 2024-03-29T11:30:02.963 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.962+0000 7f4ee15eb640 1 -- 172.21.15.67:0/2949204243 wait complete. 2024-03-29T11:30:02.963 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.963+0000 7f4ee15eb640 10 monclient: build_initial_monmap 2024-03-29T11:30:02.963 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.963+0000 7f4ee15eb640 10 monclient: monmap: 2024-03-29T11:30:02.963 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:30:02.964 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:02.964 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:30:02.964 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:30:02.964 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:02.964 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:02.964 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:30:02.965 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:30:02.965 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:30:02.965 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:02.965 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.964+0000 7f4ee15eb640 1 librados: starting msgr at 2024-03-29T11:30:02.965 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.964+0000 7f4ee15eb640 1 librados: starting objecter 2024-03-29T11:30:02.965 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.964+0000 7f4ee15eb640 1 Processor -- start 2024-03-29T11:30:02.965 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.964+0000 7f4ee15eb640 1 -- start start 2024-03-29T11:30:02.965 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.964+0000 7f4ee15eb640 1 librados: setting wanted keys 2024-03-29T11:30:02.965 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.964+0000 7f4ee15eb640 1 librados: calling monclient init 2024-03-29T11:30:02.965 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.964+0000 7f4ee15eb640 10 monclient: init 2024-03-29T11:30:02.965 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.965+0000 7f4ee15eb640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:02.965 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.965+0000 7f4ee15eb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:02.966 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.966+0000 7f4ee15eb640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed4006410 addr v1:172.21.15.67:6790/0 2024-03-29T11:30:02.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.966+0000 7f4ee15eb640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed40327d0 addr v1:172.21.15.82:6789/0 2024-03-29T11:30:02.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.966+0000 7f4ee15eb640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed4034850 addr v1:172.21.15.67:6789/0 2024-03-29T11:30:02.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.966+0000 7f4ee15eb640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed40c2c90 con 0x7f4ed4034850 2024-03-29T11:30:02.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.966+0000 7f4ee15eb640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed40c3e90 con 0x7f4ed40327d0 2024-03-29T11:30:02.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.966+0000 7f4ee15eb640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed40c3370 con 0x7f4ed4006410 2024-03-29T11:30:02.973 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.966+0000 7f4ee15eb640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:02.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.968+0000 7f4edbfff640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7f4ed4006410 0x7f4ed40bad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:56288/0 (socket says 172.21.15.67:56288) 2024-03-29T11:30:02.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.968+0000 7f4edbfff640 1 -- 172.21.15.67:0/1540480516 learned_addr learned my addr 172.21.15.67:0/1540480516 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:30:02.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.970+0000 7f4ee15eb640 10 monclient(hunting): authenticate will time out at 1463.265991s 2024-03-29T11:30:02.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.970+0000 7f4ed8ff9640 10 client.?.objecter ms_handle_connect 0x7f4ed4006410 2024-03-29T11:30:02.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.970+0000 7f4ed8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:02.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.970+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169049809 0 0) 0x7f4ed40c3370 con 0x7f4ed4006410 2024-03-29T11:30:02.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.970+0000 7f4ed8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:02.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.970+0000 7f4ed8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:02.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.970+0000 7f4ed8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:02.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.970+0000 7f4ed8ff9640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:30:02.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.970+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec0004eb0 con 0x7f4ed4006410 2024-03-29T11:30:02.974 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.971+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2612307259 0 0) 0x7f4ec0004eb0 con 0x7f4ed4006410 2024-03-29T11:30:02.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.971+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ed40c3370 con 0x7f4ed4006410 2024-03-29T11:30:02.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.971+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f4ed0005e70 con 0x7f4ed4006410 2024-03-29T11:30:02.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.971+0000 7f4ed8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:02.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.971+0000 7f4ed8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:02.975 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.971+0000 7f4ed8ff9640 10 monclient(hunting): dump: 2024-03-29T11:30:02.975 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:02.975 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:02.975 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:02.975 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:02.975 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:02.975 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:02.975 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:02.976 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:02.976 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:02.976 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:02.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.972+0000 7f4ed8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:02.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.972+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3006102174 0 0) 0x7f4ed40c3370 con 0x7f4ed4006410 2024-03-29T11:30:02.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.972+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 >> v1:172.21.15.82:6789/0 conn(0x7f4ed40327d0 legacy=0x7f4ed4158400 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:30:02.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.972+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 >> v1:172.21.15.67:6789/0 conn(0x7f4ed4034850 legacy=0x7f4ed4158b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:02.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.973+0000 7f4ed8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:30:02.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.973+0000 7f4ed8ff9640 1 monclient: found mon.c 2024-03-29T11:30:02.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.973+0000 7f4ed8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:02.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.973+0000 7f4ed8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:02.976 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.973+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed40c3500 con 0x7f4ed4006410 2024-03-29T11:30:02.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.973+0000 7f4ed8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:30:02.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.973+0000 7f4ed8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:30:02.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ed8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:02.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ee15eb640 5 monclient: authenticate success, global_id 4649 2024-03-29T11:30:02.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ee15eb640 10 monclient: _renew_subs 2024-03-29T11:30:02.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ee15eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:02.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ee15eb640 1 -- 172.21.15.67:0/1540480516 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed40bf8d0 con 0x7f4ed4006410 2024-03-29T11:30:02.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ed8ff9640 10 client.?.objecter ms_handle_connect 0x7f4ed4034850 2024-03-29T11:30:02.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ee15eb640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:02.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ee15eb640 10 monclient: _renew_subs 2024-03-29T11:30:02.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ee15eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:02.977 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ee15eb640 1 -- 172.21.15.67:0/1540480516 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed40bb470 con 0x7f4ed4006410 2024-03-29T11:30:02.978 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ee15eb640 1 librados: init done 2024-03-29T11:30:02.978 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ed8ff9640 10 client.4649.objecter resend_mon_ops 2024-03-29T11:30:02.978 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ed0001600 con 0x7f4ed4006410 2024-03-29T11:30:02.978 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ed8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:02.978 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.974+0000 7f4ed9ffb640 4 set_mon_vals no callback set 2024-03-29T11:30:02.978 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.975+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f4ed0001a50 con 0x7f4ed4006410 2024-03-29T11:30:02.978 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.975+0000 7f4ed8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:02.978 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.975+0000 7f4ed8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:02.979 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.975+0000 7f4ed8ff9640 10 monclient: dump: 2024-03-29T11:30:02.979 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:02.979 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:02.979 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:02.979 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:02.979 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:02.979 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:02.979 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:02.980 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:02.980 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:02.980 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:02.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.977+0000 7f4ee15eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:02.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.977+0000 7f4ee15eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:02.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.977+0000 7f4ee15eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:02.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.977+0000 7f4ee15eb640 1 -- 172.21.15.67:0/1540480516 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ebc005080 con 0x7f4ed4006410 2024-03-29T11:30:02.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.977+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f4ed0001bb0 con 0x7f4ed4006410 2024-03-29T11:30:02.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.978+0000 7f4edb7fe640 10 monclient: get_auth_request con 0x7f4ec006dba0 auth_method 0 2024-03-29T11:30:02.980 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.979+0000 7f4ed8ff9640 10 client.4649.objecter ms_handle_connect 0x7f4ec006dba0 2024-03-29T11:30:02.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.979+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3315246757 0 0) 0x7f4ed0084aa0 con 0x7f4ed4006410 2024-03-29T11:30:02.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.979+0000 7f4ed8ff9640 10 client.4649.objecter ms_dispatch 0x7f4edc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:02.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.979+0000 7f4ed8ff9640 3 client.4649.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:02.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.979+0000 7f4ed8ff9640 3 client.4649.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:02.981 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.979+0000 7f4ed8ff9640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-29T11:30:02.982 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.988+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ed0050380 con 0x7f4ed4006410 2024-03-29T11:30:02.982 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.988+0000 7f4ed8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:02.982 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:02.988+0000 7f4ed8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:03.314 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.321+0000 7f4ee15eb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:03.314 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.321+0000 7f4ee15eb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:03.314 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.321+0000 7f4ee15eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:03.314 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.321+0000 7f4ee15eb640 1 -- 172.21.15.67:0/1540480516 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ebc005330 con 0x7f4ed4006410 2024-03-29T11:30:03.315 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.323+0000 7f4ed8ff9640 1 -- 172.21.15.67:0/1540480516 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1631012267) 0x7f4ed00522b0 con 0x7f4ed4006410 2024-03-29T11:30:03.315 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.323+0000 7f4ed8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:03.315 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.323+0000 7f4ed8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:03.315 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:30:03.316 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":11,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:29:31.896474+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:31.896474+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:03.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.326+0000 7f4ee15eb640 10 librados: watch_flush enter 2024-03-29T11:30:03.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.327+0000 7f4ee15eb640 10 librados: watch_flush exit 2024-03-29T11:30:03.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.327+0000 7f4ee15eb640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-29T11:30:03.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.327+0000 7f4ee15eb640 10 client.4649.objecter successfully canceled tick 2024-03-29T11:30:03.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.327+0000 7f4ee15eb640 1 -- 172.21.15.67:0/1540480516 >> v1:172.21.15.67:6816/404787197 conn(0x7f4ec006dba0 legacy=0x7f4ec0070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.327+0000 7f4ee15eb640 10 monclient: shutdown 2024-03-29T11:30:03.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.327+0000 7f4ee15eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:03.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.327+0000 7f4ee15eb640 1 -- 172.21.15.67:0/1540480516 >> v1:172.21.15.67:6790/0 conn(0x7f4ed4006410 legacy=0x7f4ed40bad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.327+0000 7f4ee15eb640 1 -- 172.21.15.67:0/1540480516 shutdown_connections 2024-03-29T11:30:03.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.327+0000 7f4ee15eb640 1 -- 172.21.15.67:0/1540480516 >> 172.21.15.67:0/1540480516 conn(0x7f4ed40058f0 msgr2=0x7f4ed402a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:03.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.328+0000 7f4ee15eb640 1 -- 172.21.15.67:0/1540480516 shutdown_connections 2024-03-29T11:30:03.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.328+0000 7f4ee15eb640 1 -- 172.21.15.67:0/1540480516 wait complete. 2024-03-29T11:30:03.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:03.328+0000 7f4ee15eb640 1 librados: shutdown 2024-03-29T11:30:03.338 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:10.147 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:10.311 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.318+0000 7f94313cf640 10 monclient: get_monmap_and_config 2024-03-29T11:30:10.311 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.318+0000 7f94313cf640 10 monclient: build_initial_monmap 2024-03-29T11:30:10.311 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.318+0000 7f94313cf640 10 monclient: monmap: 2024-03-29T11:30:10.311 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:30:10.311 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:10.311 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:30:10.319018+0000 2024-03-29T11:30:10.312 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:30:10.319018+0000 2024-03-29T11:30:10.312 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:10.312 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:10.312 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:30:10.312 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:30:10.312 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:30:10.312 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:10.313 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.320+0000 7f94313cf640 1 Processor -- start 2024-03-29T11:30:10.313 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.320+0000 7f94313cf640 1 -- start start 2024-03-29T11:30:10.313 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.320+0000 7f94313cf640 10 monclient: init 2024-03-29T11:30:10.314 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.321+0000 7f94313cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:10.314 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.321+0000 7f94313cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:10.314 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.321+0000 7f94313cf640 10 monclient(hunting): picked mon.noname-b con 0x7f942c0a9930 addr v1:172.21.15.67:6790/0 2024-03-29T11:30:10.314 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.321+0000 7f94313cf640 10 monclient(hunting): picked mon.noname-a con 0x7f942c149dc0 addr v1:172.21.15.67:6789/0 2024-03-29T11:30:10.314 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.321+0000 7f94313cf640 10 monclient(hunting): picked mon.noname-c con 0x7f942c14ff40 addr v1:172.21.15.82:6789/0 2024-03-29T11:30:10.314 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.321+0000 7f94313cf640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f942c0aae60 con 0x7f942c149dc0 2024-03-29T11:30:10.314 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.321+0000 7f94313cf640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f942c0b4310 con 0x7f942c14ff40 2024-03-29T11:30:10.314 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.321+0000 7f94313cf640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f942c0afdd0 con 0x7f942c0a9930 2024-03-29T11:30:10.314 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.321+0000 7f94313cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:10.315 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.321+0000 7f94313cf640 10 monclient(hunting): authenticate will time out at 1470.617527s 2024-03-29T11:30:10.317 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.325+0000 7f942a7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f942c149dc0 0x7f942c14a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56775/0 (socket says 172.21.15.67:56775) 2024-03-29T11:30:10.318 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.325+0000 7f942a7fc640 1 -- 172.21.15.67:0/3798394058 learned_addr learned my addr 172.21.15.67:0/3798394058 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:30:10.319 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.326+0000 7f9429ffb640 1 -- 172.21.15.67:0/3798394058 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827295702 0 0) 0x7f942c0aae60 con 0x7f942c149dc0 2024-03-29T11:30:10.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.326+0000 7f9429ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:10.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.326+0000 7f9429ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:10.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.326+0000 7f9429ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:10.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.326+0000 7f9429ffb640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:30:10.320 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.326+0000 7f9429ffb640 1 -- 172.21.15.67:0/3798394058 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9414003520 con 0x7f942c149dc0 2024-03-29T11:30:10.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 1 -- 172.21.15.67:0/3798394058 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4022917192 0 0) 0x7f9414003520 con 0x7f942c149dc0 2024-03-29T11:30:10.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 1 -- 172.21.15.67:0/3798394058 >> v1:172.21.15.67:6790/0 conn(0x7f942c0a9930 legacy=0x7f942c143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:10.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 1 -- 172.21.15.67:0/3798394058 >> v1:172.21.15.82:6789/0 conn(0x7f942c14ff40 legacy=0x7f942c152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:10.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 1 monclient: found mon.noname-a 2024-03-29T11:30:10.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:10.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:30:10.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 1 -- 172.21.15.67:0/3798394058 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942c14a950 con 0x7f942c149dc0 2024-03-29T11:30:10.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:10.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:10.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:10.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 1 -- 172.21.15.67:0/3798394058 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f9420001690 con 0x7f942c149dc0 2024-03-29T11:30:10.322 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:10.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f9429ffb640 10 monclient: dump: 2024-03-29T11:30:10.323 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:10.323 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:10.323 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:10.323 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:10.323 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.323 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:10.323 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:10.323 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:10.323 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:10.323 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:10.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f94313cf640 5 monclient: authenticate success, global_id 4590 2024-03-29T11:30:10.323 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.329+0000 7f94313cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:10.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.330+0000 7f9429ffb640 1 -- 172.21.15.67:0/3798394058 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9420001860 con 0x7f942c149dc0 2024-03-29T11:30:10.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.330+0000 7f9429ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:10.324 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.330+0000 7f9429ffb640 1 -- 172.21.15.67:0/3798394058 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f9420001cb0 con 0x7f942c149dc0 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.331+0000 7f9429ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.331+0000 7f9429ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.331+0000 7f9429ffb640 10 monclient: dump: 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.331+0000 7f94313cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:10.325 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.331+0000 7f94313cf640 4 set_mon_vals no callback set 2024-03-29T11:30:10.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.332+0000 7f94313cf640 10 monclient: shutdown 2024-03-29T11:30:10.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.332+0000 7f94313cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:10.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.332+0000 7f94313cf640 1 -- 172.21.15.67:0/3798394058 >> v1:172.21.15.67:6789/0 conn(0x7f942c149dc0 legacy=0x7f942c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.332+0000 7f94313cf640 1 -- 172.21.15.67:0/3798394058 shutdown_connections 2024-03-29T11:30:10.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.332+0000 7f94313cf640 1 -- 172.21.15.67:0/3798394058 >> 172.21.15.67:0/3798394058 conn(0x7f942c0b35e0 msgr2=0x7f942c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:10.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.332+0000 7f94313cf640 1 -- 172.21.15.67:0/3798394058 shutdown_connections 2024-03-29T11:30:10.326 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.332+0000 7f94313cf640 1 -- 172.21.15.67:0/3798394058 wait complete. 2024-03-29T11:30:10.329 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.333+0000 7f94313cf640 10 monclient: build_initial_monmap 2024-03-29T11:30:10.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.333+0000 7f94313cf640 10 monclient: monmap: 2024-03-29T11:30:10.330 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:30:10.330 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:10.330 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:30:10.330 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:30:10.330 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:10.330 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:10.330 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:30:10.330 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:30:10.330 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:30:10.330 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:10.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.333+0000 7f94313cf640 1 librados: starting msgr at 2024-03-29T11:30:10.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.333+0000 7f94313cf640 1 librados: starting objecter 2024-03-29T11:30:10.330 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.334+0000 7f94313cf640 1 Processor -- start 2024-03-29T11:30:10.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.334+0000 7f94313cf640 1 -- start start 2024-03-29T11:30:10.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.334+0000 7f94313cf640 1 librados: setting wanted keys 2024-03-29T11:30:10.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.334+0000 7f94313cf640 1 librados: calling monclient init 2024-03-29T11:30:10.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.334+0000 7f94313cf640 10 monclient: init 2024-03-29T11:30:10.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.335+0000 7f94313cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:10.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.335+0000 7f94313cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:10.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.335+0000 7f94313cf640 10 monclient(hunting): picked mon.noname-c con 0x7f942c0a9930 addr v1:172.21.15.67:6790/0 2024-03-29T11:30:10.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.335+0000 7f94313cf640 10 monclient(hunting): picked mon.noname-a con 0x7f942c149dc0 addr v1:172.21.15.67:6789/0 2024-03-29T11:30:10.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.335+0000 7f94313cf640 10 monclient(hunting): picked mon.noname-b con 0x7f942c14ff40 addr v1:172.21.15.82:6789/0 2024-03-29T11:30:10.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.335+0000 7f94313cf640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f942c1490f0 con 0x7f942c149dc0 2024-03-29T11:30:10.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.335+0000 7f94313cf640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f942c144c70 con 0x7f942c14ff40 2024-03-29T11:30:10.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.335+0000 7f94313cf640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f942c1dc250 con 0x7f942c0a9930 2024-03-29T11:30:10.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.335+0000 7f94313cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:10.331 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.335+0000 7f94313cf640 10 monclient(hunting): authenticate will time out at 1470.631334s 2024-03-29T11:30:10.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.336+0000 7f942b7fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f942c14ff40 0x7f942c1e5ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:53512/0 (socket says 172.21.15.67:53512) 2024-03-29T11:30:10.332 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.336+0000 7f942b7fe640 1 -- 172.21.15.67:0/4261306163 learned_addr learned my addr 172.21.15.67:0/4261306163 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:30:10.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.338+0000 7f94137fe640 10 client.?.objecter ms_handle_connect 0x7f942c14ff40 2024-03-29T11:30:10.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.338+0000 7f94137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:10.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.338+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841180876 0 0) 0x7f942c144c70 con 0x7f942c14ff40 2024-03-29T11:30:10.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.338+0000 7f94137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:10.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.338+0000 7f94137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:10.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.338+0000 7f94137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:10.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.338+0000 7f94137fe640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:30:10.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.338+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9418004eb0 con 0x7f942c14ff40 2024-03-29T11:30:10.336 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.339+0000 7f94137fe640 10 client.?.objecter ms_handle_connect 0x7f942c0a9930 2024-03-29T11:30:10.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.339+0000 7f94137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:10.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.339+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224711122 0 0) 0x7f942c1dc250 con 0x7f942c0a9930 2024-03-29T11:30:10.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.339+0000 7f94137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:10.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.339+0000 7f94137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:10.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.339+0000 7f94137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:10.337 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.339+0000 7f94137fe640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:30:10.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.339+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94180051d0 con 0x7f942c0a9930 2024-03-29T11:30:10.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.340+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1881722512 0 0) 0x7f9418004eb0 con 0x7f942c14ff40 2024-03-29T11:30:10.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.340+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f942c1dc250 con 0x7f942c14ff40 2024-03-29T11:30:10.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.340+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f941c001850 con 0x7f942c14ff40 2024-03-29T11:30:10.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.340+0000 7f94137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.340+0000 7f94137fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:10.338 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.340+0000 7f94137fe640 10 monclient(hunting): dump: 2024-03-29T11:30:10.338 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:10.338 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:10.338 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:10.338 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:10.338 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.339 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:10.339 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:10.339 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:10.339 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:10.339 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:10.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.340+0000 7f94137fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:10.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3100989459 0 0) 0x7f942c1dc250 con 0x7f942c14ff40 2024-03-29T11:30:10.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 >> v1:172.21.15.67:6790/0 conn(0x7f942c0a9930 legacy=0x7f942c1489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 >> v1:172.21.15.67:6789/0 conn(0x7f942c149dc0 legacy=0x7f942c1dbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94137fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:10.339 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94137fe640 1 monclient: found mon.b 2024-03-29T11:30:10.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:10.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94137fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:30:10.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942c1414c0 con 0x7f942c14ff40 2024-03-29T11:30:10.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94137fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:10.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94137fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:10.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:10.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94313cf640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:30:10.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94313cf640 10 monclient: _renew_subs 2024-03-29T11:30:10.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94313cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:30:10.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94313cf640 1 -- 172.21.15.67:0/4261306163 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f942c1416d0 con 0x7f942c14ff40 2024-03-29T11:30:10.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94313cf640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:10.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94313cf640 10 monclient: _renew_subs 2024-03-29T11:30:10.340 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94313cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:30:10.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94313cf640 1 -- 172.21.15.67:0/4261306163 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f942c147eb0 con 0x7f942c14ff40 2024-03-29T11:30:10.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.342+0000 7f94313cf640 1 librados: init done 2024-03-29T11:30:10.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.343+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f941c001d40 con 0x7f942c14ff40 2024-03-29T11:30:10.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.343+0000 7f94137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:10.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.343+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f941c0037b0 con 0x7f942c14ff40 2024-03-29T11:30:10.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.343+0000 7f94137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.343+0000 7f94137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:10.341 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.343+0000 7f94137fe640 10 monclient: dump: 2024-03-29T11:30:10.341 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:10.341 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:10.341 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:10.341 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:10.341 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.342 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:10.342 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:10.342 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:10.342 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:10.342 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:10.342 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.343+0000 7f94297fa640 4 set_mon_vals no callback set 2024-03-29T11:30:10.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.345+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f941c003990 con 0x7f942c14ff40 2024-03-29T11:30:10.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.348+0000 7f94313cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:10.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.348+0000 7f94313cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:10.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.348+0000 7f94313cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:30:10.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.348+0000 7f94313cf640 1 -- 172.21.15.67:0/4261306163 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f940c005080 con 0x7f942c14ff40 2024-03-29T11:30:10.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.350+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3315246757 0 0) 0x7f941c0848a0 con 0x7f942c14ff40 2024-03-29T11:30:10.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.350+0000 7f94137fe640 10 client.4546.objecter ms_dispatch 0x7f942c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:10.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.350+0000 7f94137fe640 3 client.4546.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:10.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.350+0000 7f94137fe640 3 client.4546.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:10.343 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.350+0000 7f94137fe640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T11:30:10.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.354+0000 7f942affd640 10 monclient: get_auth_request con 0x7f941806ddc0 auth_method 0 2024-03-29T11:30:10.346 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.354+0000 7f94137fe640 10 client.4546.objecter ms_handle_connect 0x7f941806ddc0 2024-03-29T11:30:10.347 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.356+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f941c0501e0 con 0x7f942c14ff40 2024-03-29T11:30:10.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.356+0000 7f94137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:10.348 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.356+0000 7f94137fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:10.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.691+0000 7f94313cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:10.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.691+0000 7f94313cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:10.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.691+0000 7f94313cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T11:30:10.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.691+0000 7f94313cf640 1 -- 172.21.15.67:0/4261306163 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f940c005330 con 0x7f942c14ff40 2024-03-29T11:30:10.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.692+0000 7f94137fe640 1 -- 172.21.15.67:0/4261306163 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1631012267) 0x7f941c0501e0 con 0x7f942c14ff40 2024-03-29T11:30:10.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.692+0000 7f94137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:10.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.692+0000 7f94137fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:10.687 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:30:10.687 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":11,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:29:31.896474+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:31.896474+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:10.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.698+0000 7f94313cf640 10 librados: watch_flush enter 2024-03-29T11:30:10.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.698+0000 7f94313cf640 10 librados: watch_flush exit 2024-03-29T11:30:10.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.698+0000 7f94313cf640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T11:30:10.690 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.698+0000 7f94313cf640 10 client.4546.objecter successfully canceled tick 2024-03-29T11:30:10.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.698+0000 7f94313cf640 1 -- 172.21.15.67:0/4261306163 >> v1:172.21.15.67:6816/404787197 conn(0x7f941806ddc0 legacy=0x7f9418070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.698+0000 7f94313cf640 10 monclient: shutdown 2024-03-29T11:30:10.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.698+0000 7f94313cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:10.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.698+0000 7f94313cf640 1 -- 172.21.15.67:0/4261306163 >> v1:172.21.15.82:6789/0 conn(0x7f942c14ff40 legacy=0x7f942c1e5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.700+0000 7f94313cf640 1 -- 172.21.15.67:0/4261306163 shutdown_connections 2024-03-29T11:30:10.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.700+0000 7f94313cf640 1 -- 172.21.15.67:0/4261306163 >> 172.21.15.67:0/4261306163 conn(0x7f942c0b35e0 msgr2=0x7f942c0b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:10.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.700+0000 7f94313cf640 1 -- 172.21.15.67:0/4261306163 shutdown_connections 2024-03-29T11:30:10.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.700+0000 7f94313cf640 1 -- 172.21.15.67:0/4261306163 wait complete. 2024-03-29T11:30:10.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:10.700+0000 7f94313cf640 1 librados: shutdown 2024-03-29T11:30:10.709 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:17.516 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:17.667 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.674+0000 7fa189247640 10 monclient: get_monmap_and_config 2024-03-29T11:30:17.667 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.674+0000 7fa189247640 10 monclient: build_initial_monmap 2024-03-29T11:30:17.667 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.674+0000 7fa189247640 10 monclient: monmap: 2024-03-29T11:30:17.667 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:30:17.667 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:17.667 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:30:17.675626+0000 2024-03-29T11:30:17.667 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:30:17.675626+0000 2024-03-29T11:30:17.667 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:17.667 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:17.668 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:30:17.668 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:30:17.668 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:30:17.668 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:17.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.675+0000 7fa189247640 1 Processor -- start 2024-03-29T11:30:17.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.675+0000 7fa189247640 1 -- start start 2024-03-29T11:30:17.670 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.675+0000 7fa189247640 10 monclient: init 2024-03-29T11:30:17.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.676+0000 7fa189247640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:17.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.676+0000 7fa189247640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:17.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.676+0000 7fa189247640 10 monclient(hunting): picked mon.noname-b con 0x7fa1841457f0 addr v1:172.21.15.67:6790/0 2024-03-29T11:30:17.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.676+0000 7fa189247640 10 monclient(hunting): picked mon.noname-a con 0x7fa18413f650 addr v1:172.21.15.67:6789/0 2024-03-29T11:30:17.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.676+0000 7fa189247640 10 monclient(hunting): picked mon.noname-c con 0x7fa1840a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T11:30:17.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.676+0000 7fa189247640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1840aaef0 con 0x7fa18413f650 2024-03-29T11:30:17.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.676+0000 7fa189247640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa184145bd0 con 0x7fa1840a9d70 2024-03-29T11:30:17.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.676+0000 7fa189247640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa184145eb0 con 0x7fa1841457f0 2024-03-29T11:30:17.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.676+0000 7fa189247640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:17.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.676+0000 7fa189247640 10 monclient(hunting): authenticate will time out at 1477.972583s 2024-03-29T11:30:17.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.677+0000 7fa182d9d640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fa1841457f0 0x7fa18413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:41610/0 (socket says 172.21.15.67:41610) 2024-03-29T11:30:17.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.677+0000 7fa182d9d640 1 -- 172.21.15.67:0/40471354 learned_addr learned my addr 172.21.15.67:0/40471354 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:30:17.671 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.677+0000 7fa181d9b640 1 -- 172.21.15.67:0/40471354 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377279905 0 0) 0x7fa184145bd0 con 0x7fa1840a9d70 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.677+0000 7fa181d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.677+0000 7fa181d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.677+0000 7fa181d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.677+0000 7fa181d9b640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.677+0000 7fa181d9b640 1 -- 172.21.15.67:0/40471354 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa16c003520 con 0x7fa1840a9d70 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.678+0000 7fa181d9b640 1 -- 172.21.15.67:0/40471354 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595233273 0 0) 0x7fa184145eb0 con 0x7fa1841457f0 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.678+0000 7fa181d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.678+0000 7fa181d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.678+0000 7fa181d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.678+0000 7fa181d9b640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.678+0000 7fa181d9b640 1 -- 172.21.15.67:0/40471354 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa184145bd0 con 0x7fa1841457f0 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.678+0000 7fa181d9b640 1 -- 172.21.15.67:0/40471354 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907763664 0 0) 0x7fa1840aaef0 con 0x7fa18413f650 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.678+0000 7fa181d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.678+0000 7fa181d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:17.672 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.678+0000 7fa181d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:17.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.678+0000 7fa181d9b640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:30:17.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.678+0000 7fa181d9b640 1 -- 172.21.15.67:0/40471354 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa184145eb0 con 0x7fa18413f650 2024-03-29T11:30:17.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 1 -- 172.21.15.67:0/40471354 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2707951875 0 0) 0x7fa184145eb0 con 0x7fa18413f650 2024-03-29T11:30:17.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 1 -- 172.21.15.67:0/40471354 >> v1:172.21.15.67:6790/0 conn(0x7fa1841457f0 legacy=0x7fa18413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:17.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 1 -- 172.21.15.67:0/40471354 >> v1:172.21.15.82:6789/0 conn(0x7fa1840a9d70 legacy=0x7fa18414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:17.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:30:17.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 1 monclient: found mon.noname-a 2024-03-29T11:30:17.679 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:17.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:30:17.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 1 -- 172.21.15.67:0/40471354 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa184146340 con 0x7fa18413f650 2024-03-29T11:30:17.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 10 monclient: _finish_auth 0 2024-03-29T11:30:17.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:30:17.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:17.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 1 -- 172.21.15.67:0/40471354 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa17c001930 con 0x7fa18413f650 2024-03-29T11:30:17.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:17.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:17.680 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 10 monclient: dump: 2024-03-29T11:30:17.680 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:17.680 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:17.680 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:17.680 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:17.680 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:17.681 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:17.681 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:17.681 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:17.681 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:17.681 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:17.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa189247640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:30:17.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa189247640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:17.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 1 -- 172.21.15.67:0/40471354 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa17c001a90 con 0x7fa18413f650 2024-03-29T11:30:17.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:17.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 1 -- 172.21.15.67:0/40471354 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa17c001ee0 con 0x7fa18413f650 2024-03-29T11:30:17.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa189247640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:17.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa189247640 4 set_mon_vals no callback set 2024-03-29T11:30:17.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:17.681 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:17.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.679+0000 7fa181d9b640 10 monclient: dump: 2024-03-29T11:30:17.682 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:17.682 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:17.682 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:17.682 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:17.682 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:17.682 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:17.682 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:17.682 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:17.682 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:17.682 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:17.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.680+0000 7fa189247640 10 monclient: shutdown 2024-03-29T11:30:17.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.680+0000 7fa189247640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:17.682 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.680+0000 7fa189247640 1 -- 172.21.15.67:0/40471354 >> v1:172.21.15.67:6789/0 conn(0x7fa18413f650 legacy=0x7fa1840b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:17.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.680+0000 7fa189247640 1 -- 172.21.15.67:0/40471354 shutdown_connections 2024-03-29T11:30:17.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.680+0000 7fa189247640 1 -- 172.21.15.67:0/40471354 >> 172.21.15.67:0/40471354 conn(0x7fa184066a00 msgr2=0x7fa184066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:17.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.680+0000 7fa189247640 1 -- 172.21.15.67:0/40471354 shutdown_connections 2024-03-29T11:30:17.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.681+0000 7fa189247640 1 -- 172.21.15.67:0/40471354 wait complete. 2024-03-29T11:30:17.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.682+0000 7fa189247640 10 monclient: build_initial_monmap 2024-03-29T11:30:17.683 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.682+0000 7fa189247640 10 monclient: monmap: 2024-03-29T11:30:17.683 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:30:17.683 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:17.683 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:30:17.683 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:30:17.683 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:17.683 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:17.683 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:30:17.683 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.682+0000 7fa189247640 1 librados: starting msgr at 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.682+0000 7fa189247640 1 librados: starting objecter 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.682+0000 7fa189247640 1 Processor -- start 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.682+0000 7fa189247640 1 -- start start 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.683+0000 7fa189247640 1 librados: setting wanted keys 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.683+0000 7fa189247640 1 librados: calling monclient init 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.683+0000 7fa189247640 10 monclient: init 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.683+0000 7fa189247640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.683+0000 7fa189247640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.683+0000 7fa189247640 10 monclient(hunting): picked mon.noname-c con 0x7fa1840a9d70 addr v1:172.21.15.67:6790/0 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa189247640 10 monclient(hunting): picked mon.noname-b con 0x7fa18413f650 addr v1:172.21.15.82:6789/0 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa182d9d640 1 --1- >> v1:172.21.15.67:6790/0 conn(0x7fa1840a9d70 0x7fa1841dadf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6790/0 says I am v1:172.21.15.67:42652/0 (socket says 172.21.15.67:42652) 2024-03-29T11:30:17.684 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa182d9d640 1 -- 172.21.15.67:0/3304839604 learned_addr learned my addr 172.21.15.67:0/3304839604 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:30:17.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa16b7fe640 10 client.?.objecter ms_handle_connect 0x7fa1840a9d70 2024-03-29T11:30:17.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa16b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:17.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa189247640 10 monclient(hunting): picked mon.noname-a con 0x7fa1841457f0 addr v1:172.21.15.67:6789/0 2024-03-29T11:30:17.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa189247640 1 -- 172.21.15.67:0/3304839604 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1841da960 con 0x7fa1841457f0 2024-03-29T11:30:17.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa189247640 1 -- 172.21.15.67:0/3304839604 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1841359f0 con 0x7fa18413f650 2024-03-29T11:30:17.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa189247640 1 -- 172.21.15.67:0/3304839604 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa184140dd0 con 0x7fa1840a9d70 2024-03-29T11:30:17.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa189247640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:17.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa189247640 10 monclient(hunting): authenticate will time out at 1477.980442s 2024-03-29T11:30:17.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa16b7fe640 10 client.?.objecter ms_handle_connect 0x7fa1841457f0 2024-03-29T11:30:17.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa16b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:17.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa16b7fe640 10 client.?.objecter ms_handle_connect 0x7fa18413f650 2024-03-29T11:30:17.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.684+0000 7fa16b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:17.685 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180906986 0 0) 0x7fa184140dd0 con 0x7fa1840a9d70 2024-03-29T11:30:17.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:17.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:17.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:17.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:30:17.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa170004eb0 con 0x7fa1840a9d70 2024-03-29T11:30:17.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792576321 0 0) 0x7fa1841359f0 con 0x7fa18413f650 2024-03-29T11:30:17.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:17.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:17.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:17.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:30:17.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa184140dd0 con 0x7fa18413f650 2024-03-29T11:30:17.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959293923 0 0) 0x7fa1841da960 con 0x7fa1841457f0 2024-03-29T11:30:17.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:17.686 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:17.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:17.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:30:17.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1841359f0 con 0x7fa1841457f0 2024-03-29T11:30:17.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2615350462 0 0) 0x7fa170004eb0 con 0x7fa1840a9d70 2024-03-29T11:30:17.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.685+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1841da960 con 0x7fa1840a9d70 2024-03-29T11:30:17.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa174001950 con 0x7fa1840a9d70 2024-03-29T11:30:17.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:17.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:17.687 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 10 monclient(hunting): dump: 2024-03-29T11:30:17.687 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:17.687 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:17.687 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:17.687 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:17.687 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:17.688 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:17.688 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:17.688 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:17.688 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:17.688 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:17.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:17.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1028271546 0 0) 0x7fa1841359f0 con 0x7fa1841457f0 2024-03-29T11:30:17.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa170004eb0 con 0x7fa1841457f0 2024-03-29T11:30:17.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2936150207 0 0) 0x7fa1841da960 con 0x7fa1840a9d70 2024-03-29T11:30:17.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 >> v1:172.21.15.82:6789/0 conn(0x7fa18413f650 legacy=0x7fa1841426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:17.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 >> v1:172.21.15.67:6789/0 conn(0x7fa1841457f0 legacy=0x7fa1841e1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:17.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:17.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 1 monclient: found mon.c 2024-03-29T11:30:17.688 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:17.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:17.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa184142e00 con 0x7fa1840a9d70 2024-03-29T11:30:17.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:17.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:17.689 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa16b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:17.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa189247640 5 monclient: authenticate success, global_id 4658 2024-03-29T11:30:17.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa189247640 10 monclient: _renew_subs 2024-03-29T11:30:17.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa189247640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:17.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa189247640 1 -- 172.21.15.67:0/3304839604 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1841da510 con 0x7fa1840a9d70 2024-03-29T11:30:17.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa189247640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:17.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa189247640 10 monclient: _renew_subs 2024-03-29T11:30:17.691 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa189247640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:17.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa189247640 1 -- 172.21.15.67:0/3304839604 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1841392e0 con 0x7fa1840a9d70 2024-03-29T11:30:17.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.686+0000 7fa189247640 1 librados: init done 2024-03-29T11:30:17.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.687+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa174001ab0 con 0x7fa1840a9d70 2024-03-29T11:30:17.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.687+0000 7fa16b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:17.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.687+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7fa174004a90 con 0x7fa1840a9d70 2024-03-29T11:30:17.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.687+0000 7fa16b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:17.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.687+0000 7fa16b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:17.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.687+0000 7fa18159a640 4 set_mon_vals no callback set 2024-03-29T11:30:17.692 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.687+0000 7fa16b7fe640 10 monclient: dump: 2024-03-29T11:30:17.692 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:17.692 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:17.692 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:17.692 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:17.692 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:17.693 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:17.693 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:17.693 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:17.693 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:17.693 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:17.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.688+0000 7fa189247640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:17.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.688+0000 7fa189247640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:17.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.688+0000 7fa189247640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:17.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.688+0000 7fa189247640 1 -- 172.21.15.67:0/3304839604 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa164005080 con 0x7fa1840a9d70 2024-03-29T11:30:17.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.690+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7fa17401aa60 con 0x7fa1840a9d70 2024-03-29T11:30:17.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.691+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3315246757 0 0) 0x7fa174084980 con 0x7fa1840a9d70 2024-03-29T11:30:17.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.691+0000 7fa16b7fe640 10 client.4658.objecter ms_dispatch 0x7fa184000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:17.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.691+0000 7fa16b7fe640 3 client.4658.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:17.693 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.691+0000 7fa16b7fe640 3 client.4658.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:17.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.691+0000 7fa16b7fe640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-29T11:30:17.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.691+0000 7fa18259c640 10 monclient: get_auth_request con 0x7fa17006dd90 auth_method 0 2024-03-29T11:30:17.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.692+0000 7fa16b7fe640 10 client.4658.objecter ms_handle_connect 0x7fa17006dd90 2024-03-29T11:30:17.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.698+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa174050260 con 0x7fa1840a9d70 2024-03-29T11:30:17.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.698+0000 7fa16b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:17.694 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:17.698+0000 7fa16b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:18.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.015+0000 7fa189247640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.015+0000 7fa189247640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.015+0000 7fa189247640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:18.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.015+0000 7fa189247640 1 -- 172.21.15.67:0/3304839604 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa164005330 con 0x7fa1840a9d70 2024-03-29T11:30:18.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.016+0000 7fa16b7fe640 1 -- 172.21.15.67:0/3304839604 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1631012267) 0x7fa174052190 con 0x7fa1840a9d70 2024-03-29T11:30:18.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.016+0000 7fa16b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.016+0000 7fa16b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:18.009 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:30:18.009 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":11,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:29:31.896474+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:31.896474+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:18.012 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.019+0000 7fa189247640 10 librados: watch_flush enter 2024-03-29T11:30:18.012 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.019+0000 7fa189247640 10 librados: watch_flush exit 2024-03-29T11:30:18.012 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.019+0000 7fa189247640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-29T11:30:18.012 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.019+0000 7fa189247640 10 client.4658.objecter successfully canceled tick 2024-03-29T11:30:18.012 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.019+0000 7fa189247640 1 -- 172.21.15.67:0/3304839604 >> v1:172.21.15.67:6816/404787197 conn(0x7fa17006dd90 legacy=0x7fa170070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.019+0000 7fa189247640 10 monclient: shutdown 2024-03-29T11:30:18.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.019+0000 7fa189247640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:18.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.019+0000 7fa189247640 1 -- 172.21.15.67:0/3304839604 >> v1:172.21.15.67:6790/0 conn(0x7fa1840a9d70 legacy=0x7fa1841dadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.020+0000 7fa189247640 1 -- 172.21.15.67:0/3304839604 shutdown_connections 2024-03-29T11:30:18.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.020+0000 7fa189247640 1 -- 172.21.15.67:0/3304839604 >> 172.21.15.67:0/3304839604 conn(0x7fa184066a00 msgr2=0x7fa1840b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:18.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.020+0000 7fa189247640 1 -- 172.21.15.67:0/3304839604 shutdown_connections 2024-03-29T11:30:18.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.020+0000 7fa189247640 1 -- 172.21.15.67:0/3304839604 wait complete. 2024-03-29T11:30:18.013 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:18.020+0000 7fa189247640 1 librados: shutdown 2024-03-29T11:30:18.026 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:24.833 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:24.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.991+0000 7f33fa2f6640 10 monclient: get_monmap_and_config 2024-03-29T11:30:24.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.991+0000 7f33fa2f6640 10 monclient: build_initial_monmap 2024-03-29T11:30:24.985 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.991+0000 7f33fa2f6640 10 monclient: monmap: 2024-03-29T11:30:24.986 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:30:24.986 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:24.986 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:30:24.993646+0000 2024-03-29T11:30:24.986 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:30:24.993646+0000 2024-03-29T11:30:24.986 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:24.986 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:24.986 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:30:24.986 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T11:30:24.986 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-b 2024-03-29T11:30:24.986 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:24.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.992+0000 7f33fa2f6640 1 Processor -- start 2024-03-29T11:30:24.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.992+0000 7f33fa2f6640 1 -- start start 2024-03-29T11:30:24.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.992+0000 7f33fa2f6640 10 monclient: init 2024-03-29T11:30:24.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.994+0000 7f33fa2f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:24.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.994+0000 7f33fa2f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:24.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.994+0000 7f33fa2f6640 10 monclient(hunting): picked mon.noname-c con 0x7f33f40b4710 addr v1:172.21.15.82:6789/0 2024-03-29T11:30:24.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.994+0000 7f33fa2f6640 10 monclient(hunting): picked mon.noname-a con 0x7f33f40b3970 addr v1:172.21.15.67:6789/0 2024-03-29T11:30:24.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.994+0000 7f33fa2f6640 10 monclient(hunting): picked mon.noname-b con 0x7f33f413f390 addr v1:172.21.15.67:6790/0 2024-03-29T11:30:24.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.994+0000 7f33fa2f6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f40af020 con 0x7f33f40b3970 2024-03-29T11:30:24.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.994+0000 7f33fa2f6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f40b0910 con 0x7f33f40b4710 2024-03-29T11:30:24.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.994+0000 7f33fa2f6640 1 -- --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f40b0bf0 con 0x7f33f413f390 2024-03-29T11:30:24.988 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.994+0000 7f33fa2f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:24.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.994+0000 7f33fa2f6640 10 monclient(hunting): authenticate will time out at 1485.291543s 2024-03-29T11:30:24.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.994+0000 7f33f3fff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f33f40b4710 0x7f33f40b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.67:59446/0 (socket says 172.21.15.67:59446) 2024-03-29T11:30:24.989 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.994+0000 7f33f3fff640 1 -- 172.21.15.67:0/3656119847 learned_addr learned my addr 172.21.15.67:0/3656119847 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:30:24.990 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.995+0000 7f33f2ffd640 1 -- 172.21.15.67:0/3656119847 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905617376 0 0) 0x7f33f40af020 con 0x7f33f40b3970 2024-03-29T11:30:24.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.995+0000 7f33f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:24.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.995+0000 7f33f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:24.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.995+0000 7f33f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:24.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.995+0000 7f33f2ffd640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:30:24.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.995+0000 7f33f2ffd640 1 -- 172.21.15.67:0/3656119847 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33dc003520 con 0x7f33f40b3970 2024-03-29T11:30:24.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.995+0000 7f33f2ffd640 1 -- 172.21.15.67:0/3656119847 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360711570 0 0) 0x7f33f40b0bf0 con 0x7f33f413f390 2024-03-29T11:30:24.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.995+0000 7f33f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:24.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.995+0000 7f33f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:24.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.995+0000 7f33f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:24.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.995+0000 7f33f2ffd640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:30:24.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.995+0000 7f33f2ffd640 1 -- 172.21.15.67:0/3656119847 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33dc0048a0 con 0x7f33f413f390 2024-03-29T11:30:24.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 1 -- 172.21.15.67:0/3656119847 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1141791335 0 0) 0x7f33dc003520 con 0x7f33f40b3970 2024-03-29T11:30:24.991 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 1 -- 172.21.15.67:0/3656119847 >> v1:172.21.15.67:6790/0 conn(0x7f33f413f390 legacy=0x7f33f413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:24.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 1 -- 172.21.15.67:0/3656119847 >> v1:172.21.15.82:6789/0 conn(0x7f33f40b4710 legacy=0x7f33f40b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:24.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:24.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 1 monclient: found mon.noname-a 2024-03-29T11:30:24.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:24.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.67:6789/0 2024-03-29T11:30:24.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 1 -- 172.21.15.67:0/3656119847 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f40aeb00 con 0x7f33f40b3970 2024-03-29T11:30:24.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:24.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:24.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:24.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33fa2f6640 5 monclient: authenticate success, global_id 4599 2024-03-29T11:30:24.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33fa2f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:24.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 1 -- 172.21.15.67:0/3656119847 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f33e8001670 con 0x7f33f40b3970 2024-03-29T11:30:24.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:24.992 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 10 monclient: dump: 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33fa2f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 1 -- 172.21.15.67:0/3656119847 <== mon.0 v1:172.21.15.67:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33e8001850 con 0x7f33f40b3970 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:24.993 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33fa2f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:24.994 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.996+0000 7f33fa2f6640 4 set_mon_vals no callback set 2024-03-29T11:30:24.994 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.997+0000 7f33f2ffd640 1 -- 172.21.15.67:0/3656119847 <== mon.0 v1:172.21.15.67:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f33e8001670 con 0x7f33f40b3970 2024-03-29T11:30:24.994 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.997+0000 7f33f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:24.994 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.997+0000 7f33f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:24.994 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.997+0000 7f33f2ffd640 10 monclient: dump: 2024-03-29T11:30:24.994 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:24.994 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:24.994 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:24.994 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:24.994 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:24.994 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:24.994 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:24.994 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:24.995 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:24.995 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:25.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.997+0000 7f33fa2f6640 10 monclient: shutdown 2024-03-29T11:30:25.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.997+0000 7f33fa2f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:25.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.997+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3656119847 >> v1:172.21.15.67:6789/0 conn(0x7f33f40b3970 legacy=0x7f33f40af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.997+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3656119847 shutdown_connections 2024-03-29T11:30:25.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.998+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3656119847 >> 172.21.15.67:0/3656119847 conn(0x7f33f4066a00 msgr2=0x7f33f4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:25.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.998+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3656119847 shutdown_connections 2024-03-29T11:30:25.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.999+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3656119847 wait complete. 2024-03-29T11:30:25.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.999+0000 7f33fa2f6640 10 monclient: build_initial_monmap 2024-03-29T11:30:25.001 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:24.999+0000 7f33fa2f6640 10 monclient: monmap: 2024-03-29T11:30:25.001 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2024-03-29T11:30:25.001 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:25.002 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2024-03-29T11:30:25.002 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2024-03-29T11:30:25.002 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:25.002 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:25.002 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a 2024-03-29T11:30:25.002 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T11:30:25.002 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.noname-c 2024-03-29T11:30:25.002 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:25.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.000+0000 7f33fa2f6640 1 librados: starting msgr at 2024-03-29T11:30:25.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.000+0000 7f33fa2f6640 1 librados: starting objecter 2024-03-29T11:30:25.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.000+0000 7f33fa2f6640 1 Processor -- start 2024-03-29T11:30:25.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.000+0000 7f33fa2f6640 1 -- start start 2024-03-29T11:30:25.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.000+0000 7f33fa2f6640 1 librados: setting wanted keys 2024-03-29T11:30:25.002 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.000+0000 7f33fa2f6640 1 librados: calling monclient init 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.000+0000 7f33fa2f6640 10 monclient: init 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.001+0000 7f33fa2f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.001+0000 7f33fa2f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.001+0000 7f33fa2f6640 10 monclient(hunting): picked mon.noname-c con 0x7f33f40b4710 addr v1:172.21.15.67:6790/0 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.002+0000 7f33fa2f6640 10 monclient(hunting): picked mon.noname-a con 0x7f33f413f390 addr v1:172.21.15.67:6789/0 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.002+0000 7f33fa2f6640 10 monclient(hunting): picked mon.noname-b con 0x7f33f4138420 addr v1:172.21.15.82:6789/0 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.002+0000 7f33f37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f33f413f390 0x7f33f4135750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55232/0 (socket says 172.21.15.67:55232) 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.002+0000 7f33f37fe640 1 -- 172.21.15.67:0/3097656372 learned_addr learned my addr 172.21.15.67:0/3097656372 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.002+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3097656372 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f40aa720 con 0x7f33f413f390 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.002+0000 7f33f0ff9640 10 client.?.objecter ms_handle_connect 0x7f33f40b4710 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.002+0000 7f33f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.002+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3097656372 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f4136570 con 0x7f33f4138420 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.002+0000 7f33f0ff9640 10 client.?.objecter ms_handle_connect 0x7f33f413f390 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.002+0000 7f33f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:25.003 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.002+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3097656372 --> v1:172.21.15.67:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f40aa970 con 0x7f33f40b4710 2024-03-29T11:30:25.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.002+0000 7f33fa2f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:25.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.002+0000 7f33fa2f6640 10 monclient(hunting): authenticate will time out at 1485.299548s 2024-03-29T11:30:25.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 client.?.objecter ms_handle_connect 0x7f33f4138420 2024-03-29T11:30:25.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:25.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.2 v1:172.21.15.67:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519862077 0 0) 0x7f33f40aa970 con 0x7f33f40b4710 2024-03-29T11:30:25.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:30:25.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 --> v1:172.21.15.67:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e0004eb0 con 0x7f33f40b4710 2024-03-29T11:30:25.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108122592 0 0) 0x7f33f40aa720 con 0x7f33f413f390 2024-03-29T11:30:25.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.004 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:30:25.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f40aa970 con 0x7f33f413f390 2024-03-29T11:30:25.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288163876 0 0) 0x7f33f4136570 con 0x7f33f4138420 2024-03-29T11:30:25.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:30:25.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f40aa720 con 0x7f33f4138420 2024-03-29T11:30:25.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.2 v1:172.21.15.67:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3061908857 0 0) 0x7f33e0004eb0 con 0x7f33f40b4710 2024-03-29T11:30:25.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 --> v1:172.21.15.67:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33f4136570 con 0x7f33f40b4710 2024-03-29T11:30:25.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.2 v1:172.21.15.67:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f33e40018a0 con 0x7f33f40b4710 2024-03-29T11:30:25.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:25.005 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.003+0000 7f33f0ff9640 10 monclient(hunting): dump: 2024-03-29T11:30:25.006 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:25.006 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:25.006 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:25.006 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:25.006 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.006 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:25.006 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:25.006 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:25.006 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:25.006 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:25.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:25.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2713715537 0 0) 0x7f33f40aa970 con 0x7f33f413f390 2024-03-29T11:30:25.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 --> v1:172.21.15.67:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33e0004eb0 con 0x7f33f413f390 2024-03-29T11:30:25.006 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f33e8001d80 con 0x7f33f413f390 2024-03-29T11:30:25.007 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.007 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:25.007 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 10 monclient(hunting): dump: 2024-03-29T11:30:25.007 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:25.007 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:25.007 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:25.007 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:25.007 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.007 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:25.007 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:25.007 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:25.007 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:25.007 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:25.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2199205173 0 0) 0x7f33f40aa720 con 0x7f33f4138420 2024-03-29T11:30:25.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33f40aa970 con 0x7f33f4138420 2024-03-29T11:30:25.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.2 v1:172.21.15.67:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 977576680 0 0) 0x7f33f4136570 con 0x7f33f40b4710 2024-03-29T11:30:25.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 >> v1:172.21.15.82:6789/0 conn(0x7f33f4138420 legacy=0x7f33f4135e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 >> v1:172.21.15.67:6789/0 conn(0x7f33f413f390 legacy=0x7f33f4135750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:30:25.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 1 monclient: found mon.c 2024-03-29T11:30:25.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:25.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:25.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 --> v1:172.21.15.67:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f40aac50 con 0x7f33f40b4710 2024-03-29T11:30:25.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:30:25.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:30:25.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:25.008 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33fa2f6640 5 monclient: authenticate success, global_id 4664 2024-03-29T11:30:25.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33fa2f6640 10 monclient: _renew_subs 2024-03-29T11:30:25.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33fa2f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:25.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3097656372 --> v1:172.21.15.67:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f40aaf30 con 0x7f33f40b4710 2024-03-29T11:30:25.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33fa2f6640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:25.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33fa2f6640 10 monclient: _renew_subs 2024-03-29T11:30:25.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33fa2f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:25.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3097656372 --> v1:172.21.15.67:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f40aa210 con 0x7f33f40b4710 2024-03-29T11:30:25.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.004+0000 7f33fa2f6640 1 librados: init done 2024-03-29T11:30:25.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.005+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.2 v1:172.21.15.67:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33e40047e0 con 0x7f33f40b4710 2024-03-29T11:30:25.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.005+0000 7f33f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:25.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.005+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.2 v1:172.21.15.67:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 795099380 0 0) 0x7f33e4004950 con 0x7f33f40b4710 2024-03-29T11:30:25.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.005+0000 7f33f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.005+0000 7f33f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:25.009 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.005+0000 7f33f0ff9640 10 monclient: dump: 2024-03-29T11:30:25.010 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2024-03-29T11:30:25.010 INFO:teuthology.misc.health.smithi067.stderr:fsid 312c75bc-59e5-460a-bb56-f686dd06c6fb 2024-03-29T11:30:25.010 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:25.010 INFO:teuthology.misc.health.smithi067.stderr:created 2024-03-29T11:18:48.725056+0000 2024-03-29T11:30:25.010 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.010 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2024-03-29T11:30:25.010 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.a 2024-03-29T11:30:25.010 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T11:30:25.010 INFO:teuthology.misc.health.smithi067.stderr:2: v1:172.21.15.67:6790/0 mon.c 2024-03-29T11:30:25.010 INFO:teuthology.misc.health.smithi067.stderr: 2024-03-29T11:30:25.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.005+0000 7f33f27fc640 4 set_mon_vals no callback set 2024-03-29T11:30:25.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.006+0000 7f33fa2f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.006+0000 7f33fa2f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.010 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.006+0000 7f33fa2f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:25.011 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.006+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3097656372 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d4005080 con 0x7f33f40b4710 2024-03-29T11:30:25.011 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.007+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.2 v1:172.21.15.67:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2526030837 0 0) 0x7f33e401a920 con 0x7f33f40b4710 2024-03-29T11:30:25.011 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.008+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.2 v1:172.21.15.67:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3315246757 0 0) 0x7f33e40847e0 con 0x7f33f40b4710 2024-03-29T11:30:25.011 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.008+0000 7f33f0ff9640 10 client.4664.objecter ms_dispatch 0x7f33f4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:25.011 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.008+0000 7f33f0ff9640 3 client.4664.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:25.012 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.008+0000 7f33f0ff9640 3 client.4664.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:25.012 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.008+0000 7f33f37fe640 10 monclient: get_auth_request con 0x7f33e006e070 auth_method 0 2024-03-29T11:30:25.012 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.008+0000 7f33f0ff9640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-29T11:30:25.012 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.008+0000 7f33f0ff9640 10 client.4664.objecter ms_handle_connect 0x7f33e006e070 2024-03-29T11:30:25.012 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.017+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.2 v1:172.21.15.67:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33e40500c0 con 0x7f33f40b4710 2024-03-29T11:30:25.012 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.017+0000 7f33f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.012 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.017+0000 7f33f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:25.401 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.408+0000 7f33fa2f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:25.402 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.408+0000 7f33fa2f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:25.402 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.408+0000 7f33fa2f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.67:6790/0 2024-03-29T11:30:25.402 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.408+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3097656372 --> v1:172.21.15.67:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33d4005330 con 0x7f33f40b4710 2024-03-29T11:30:25.402 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.409+0000 7f33f0ff9640 1 -- 172.21.15.67:0/3097656372 <== mon.2 v1:172.21.15.67:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1631012267) 0x7f33e4051ff0 con 0x7f33f40b4710 2024-03-29T11:30:25.402 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.409+0000 7f33f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:25.402 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.409+0000 7f33f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:25.403 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-29T11:30:25.403 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":11,"fsid":"312c75bc-59e5-460a-bb56-f686dd06c6fb","created":"2024-03-29T11:19:26.972424+0000","modified":"2024-03-29T11:29:31.896474+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:31.896474+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"90c40155-32e1-4ef7-a4c5-1f1de26a6f4c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0269e563-f168-4fd0-9bf0-b38c6a7fb937","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"701716de-70a2-4029-b6a7-7e8dfe75f51b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c6b7be97-5c3e-4789-8d61-d0f4ae124b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f0dcbb47-e60b-4ee5-aa81-29812af8118a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e557d6a6-20ec-4600-a31d-f553df8334b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c9039c9-2b83-4f2c-8820-b5f2dd9fad50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e857aa17-61f7-45e4-824c-d5075c3ebf38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:25.406 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.412+0000 7f33fa2f6640 10 librados: watch_flush enter 2024-03-29T11:30:25.406 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.412+0000 7f33fa2f6640 10 librados: watch_flush exit 2024-03-29T11:30:25.406 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.412+0000 7f33fa2f6640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-29T11:30:25.406 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.412+0000 7f33fa2f6640 10 client.4664.objecter successfully canceled tick 2024-03-29T11:30:25.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.413+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3097656372 >> v1:172.21.15.67:6816/404787197 conn(0x7f33e006e070 legacy=0x7f33e0070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.413+0000 7f33fa2f6640 10 monclient: shutdown 2024-03-29T11:30:25.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.413+0000 7f33fa2f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:25.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.413+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3097656372 >> v1:172.21.15.67:6790/0 conn(0x7f33f40b4710 legacy=0x7f33f4136820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.413+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3097656372 shutdown_connections 2024-03-29T11:30:25.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.413+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3097656372 >> 172.21.15.67:0/3097656372 conn(0x7f33f4066a00 msgr2=0x7f33f4138fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:25.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.413+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3097656372 shutdown_connections 2024-03-29T11:30:25.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.413+0000 7f33fa2f6640 1 -- 172.21.15.67:0/3097656372 wait complete. 2024-03-29T11:30:25.407 INFO:teuthology.misc.health.smithi067.stderr:2024-03-29T11:30:25.413+0000 7f33fa2f6640 1 librados: shutdown 2024-03-29T11:30:25.422 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:25.423 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:30:25.425 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:30:25.425 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:30:25.425 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:25.460 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:30:25.460 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:30:25.460 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:25.534 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:30:25.535 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:30:25.535 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:25.602 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:30:25.602 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:30:25.603 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:25.677 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:30:25.677 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:30:25.677 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:25.717 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:30:25.717 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:30:25.717 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:25.788 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:30:25.788 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:30:25.788 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:25.865 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:30:25.866 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:30:25.866 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:25.932 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:30:25.933 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:30:25.933 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:30:25.933 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:25.957 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:30:25.957 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:30:25.957 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:25.997 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:30:25.997 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:30:25.997 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:30:25.997 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:26.009 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:30:26.009 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:30:26.009 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:26.063 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:30:26.064 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:30:26.064 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:26.075 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:30:26.075 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:30:26.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:30:26.177 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-29T11:30:26.178 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:30:27.599 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-29T11:30:27.599 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:30:27.695 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-29T11:30:27.695 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:30:27.752 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-29T11:30:27.752 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:30:27.794 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T11:30:27.794 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:30:31.709 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T11:30:31.709 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:30:31.771 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T11:30:31.771 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:30:31.844 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T11:30:31.844 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:30:31.894 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:30:31.896 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693/data/mon.a.tgz 2024-03-29T11:30:31.896 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-03-29T11:30:31.915 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.WeAodWodA3 2024-03-29T11:30:31.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.WeAodWodA3 2024-03-29T11:30:32.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.WeAodWodA3 2024-03-29T11:30:32.126 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.WeAodWodA3 is 211KB 2024-03-29T11:30:32.220 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.WeAodWodA3 2024-03-29T11:30:32.235 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693/data/mon.c.tgz 2024-03-29T11:30:32.235 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-03-29T11:30:32.289 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.3rLpHZMVJu 2024-03-29T11:30:32.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.3rLpHZMVJu 2024-03-29T11:30:32.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.3rLpHZMVJu 2024-03-29T11:30:32.475 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.3rLpHZMVJu is 212KB 2024-03-29T11:30:32.568 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.3rLpHZMVJu 2024-03-29T11:30:32.583 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693/data/mon.b.tgz 2024-03-29T11:30:32.583 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-03-29T11:30:32.599 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.bVmOOOcB4R 2024-03-29T11:30:32.599 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.bVmOOOcB4R 2024-03-29T11:30:32.715 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.bVmOOOcB4R 2024-03-29T11:30:32.799 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.bVmOOOcB4R is 212KB 2024-03-29T11:30:32.893 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.bVmOOOcB4R 2024-03-29T11:30:32.907 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:30:32.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:30:32.910 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:30:32.991 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:30:33.030 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:30:33.030 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:30:33.032 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693/remote/smithi067/crash 2024-03-29T11:30:33.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:30:33.061 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693/remote/smithi082/crash 2024-03-29T11:30:33.062 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:30:33.097 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:30:33.097 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:30:33.106 DEBUG:teuthology.orchestra.run.smithi082:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:30:33.175 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41462.log 2024-03-29T11:30:33.175 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38533.log 2024-03-29T11:30:33.175 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41462.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:30:33.175 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39976.log 2024-03-29T11:30:33.175 INFO:teuthology.orchestra.run.smithi067.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41462.log.gz 2024-03-29T11:30:33.175 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39742.log 2024-03-29T11:30:33.175 INFO:teuthology.orchestra.run.smithi067.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.38533.log.gz 2024-03-29T11:30:33.175 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40955.log 2024-03-29T11:30:33.175 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38806.log 2024-03-29T11:30:33.175 INFO:teuthology.orchestra.run.smithi067.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39976.log.gz 2024-03-29T11:30:33.175 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38257.log 2024-03-29T11:30:33.176 INFO:teuthology.orchestra.run.smithi067.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39742.log.gz 2024-03-29T11:30:33.176 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38806.log: gzip -5 --verbose -- 85.7% /var/log/ceph/ceph-client.admin.39040.log 2024-03-29T11:30:33.176 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40955.log.gz 2024-03-29T11:30:33.176 INFO:teuthology.orchestra.run.smithi067.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38806.log.gz 2024-03-29T11:30:33.176 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40370.log 2024-03-29T11:30:33.176 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41033.log 2024-03-29T11:30:33.176 INFO:teuthology.orchestra.run.smithi067.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38257.log.gz 2024-03-29T11:30:33.176 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40916.log 2024-03-29T11:30:33.176 INFO:teuthology.orchestra.run.smithi067.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39040.log.gz 2024-03-29T11:30:33.176 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41033.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40370.log.gz 2024-03-29T11:30:33.176 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40487.log 2024-03-29T11:30:33.176 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41423.log 2024-03-29T11:30:33.176 INFO:teuthology.orchestra.run.smithi067.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41033.log.gz 2024-03-29T11:30:33.176 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40487.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40916.log.gz 2024-03-29T11:30:33.177 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40838.log 2024-03-29T11:30:33.177 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41189.log 2024-03-29T11:30:33.177 INFO:teuthology.orchestra.run.smithi067.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40487.log.gz 2024-03-29T11:30:33.177 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39703.log 2024-03-29T11:30:33.177 INFO:teuthology.orchestra.run.smithi067.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41423.log.gz 2024-03-29T11:30:33.177 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39664.log 2024-03-29T11:30:33.177 INFO:teuthology.orchestra.run.smithi067.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40838.log.gz 2024-03-29T11:30:33.177 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39703.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41189.log.gzgzip 2024-03-29T11:30:33.177 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39469.log 2024-03-29T11:30:33.177 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39820.log 2024-03-29T11:30:33.177 INFO:teuthology.orchestra.run.smithi067.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39703.log.gz 2024-03-29T11:30:33.177 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39391.log 2024-03-29T11:30:33.177 INFO:teuthology.orchestra.run.smithi067.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39664.log.gz 2024-03-29T11:30:33.178 INFO:teuthology.orchestra.run.smithi067.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39469.log.gz 2024-03-29T11:30:33.178 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40994.log 2024-03-29T11:30:33.178 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40015.log 2024-03-29T11:30:33.178 INFO:teuthology.orchestra.run.smithi067.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39820.log.gz 2024-03-29T11:30:33.178 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40211.log 2024-03-29T11:30:33.178 INFO:teuthology.orchestra.run.smithi067.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39391.log.gz 2024-03-29T11:30:33.178 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40015.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40994.log.gz 2024-03-29T11:30:33.178 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:30:33.178 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41384.log 2024-03-29T11:30:33.178 INFO:teuthology.orchestra.run.smithi067.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40015.log.gz 2024-03-29T11:30:33.178 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38416.log 2024-03-29T11:30:33.178 INFO:teuthology.orchestra.run.smithi067.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40211.log.gz 2024-03-29T11:30:33.178 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38494.log 2024-03-29T11:30:33.179 INFO:teuthology.orchestra.run.smithi067.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:30:33.179 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39118.log 2024-03-29T11:30:33.179 INFO:teuthology.orchestra.run.smithi067.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41384.log.gz 2024-03-29T11:30:33.179 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40643.log 2024-03-29T11:30:33.179 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38494.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38416.log.gz 2024-03-29T11:30:33.179 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39859.log 2024-03-29T11:30:33.179 INFO:teuthology.orchestra.run.smithi067.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38494.log.gz 2024-03-29T11:30:33.179 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40643.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39118.log.gz 2024-03-29T11:30:33.179 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41306.log 2024-03-29T11:30:33.179 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38728.log 2024-03-29T11:30:33.179 INFO:teuthology.orchestra.run.smithi067.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40643.log.gz 2024-03-29T11:30:33.179 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38023.log 2024-03-29T11:30:33.179 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38728.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39859.log.gz 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34327.log 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41306.log.gz 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38298.log 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38728.log.gz 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38101.log 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38023.log.gz 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41267.log 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34327.log.gz 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38101.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38298.log.gz 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28505.log 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38140.log 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38101.log.gz 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.tmp-client.admin.28505.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28505.log.gz 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38338.log 2024-03-29T11:30:33.180 INFO:teuthology.orchestra.run.smithi067.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41267.log.gz 2024-03-29T11:30:33.181 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38611.log 2024-03-29T11:30:33.181 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38338.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:30:33.181 INFO:teuthology.orchestra.run.smithi067.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38140.log.gz 2024-03-29T11:30:33.181 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38611.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38338.log.gz 2024-03-29T11:30:33.181 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39898.log 2024-03-29T11:30:33.181 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39079.log 2024-03-29T11:30:33.181 INFO:teuthology.orchestra.run.smithi067.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38611.log.gz 2024-03-29T11:30:33.181 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41072.log 2024-03-29T11:30:33.181 INFO:teuthology.orchestra.run.smithi067.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39898.log.gz 2024-03-29T11:30:33.181 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40093.log 2024-03-29T11:30:33.181 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39235.log 2024-03-29T11:30:33.181 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41072.log: /var/log/ceph/ceph-client.admin.40093.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39079.log.gz 2024-03-29T11:30:33.182 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39274.log 2024-03-29T11:30:33.182 INFO:teuthology.orchestra.run.smithi067.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41072.log.gz 2024-03-29T11:30:33.182 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40093.log.gz 2024-03-29T11:30:33.182 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39352.log 2024-03-29T11:30:33.182 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39430.log 2024-03-29T11:30:33.182 INFO:teuthology.orchestra.run.smithi067.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39235.log.gz 2024-03-29T11:30:33.182 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38845.log 2024-03-29T11:30:33.182 INFO:teuthology.orchestra.run.smithi067.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39274.log.gz 2024-03-29T11:30:33.182 INFO:teuthology.orchestra.run.smithi067.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39352.log.gz 2024-03-29T11:30:33.182 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40799.log 2024-03-29T11:30:33.182 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39508.log 2024-03-29T11:30:33.182 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38845.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39430.log.gz 2024-03-29T11:30:33.182 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41228.log 2024-03-29T11:30:33.183 INFO:teuthology.orchestra.run.smithi067.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38845.log.gz 2024-03-29T11:30:33.183 INFO:teuthology.orchestra.run.smithi067.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.40799.log.gz/var/log/ceph/ceph-client.admin.39508.log: 2024-03-29T11:30:33.183 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:30:33.183 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40682.log 2024-03-29T11:30:33.183 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41228.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39508.log.gz 2024-03-29T11:30:33.183 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40877.log 2024-03-29T11:30:33.183 INFO:teuthology.orchestra.run.smithi067.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41228.log.gz 2024-03-29T11:30:33.183 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38962.log 2024-03-29T11:30:33.183 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39001.log 2024-03-29T11:30:33.183 INFO:teuthology.orchestra.run.smithi067.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40682.log.gz 2024-03-29T11:30:33.183 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38962.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40877.log.gz 2024-03-29T11:30:33.183 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34718.log 2024-03-29T11:30:33.183 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41345.log 2024-03-29T11:30:33.183 INFO:teuthology.orchestra.run.smithi067.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38962.log.gz 2024-03-29T11:30:33.184 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40172.log 2024-03-29T11:30:33.184 INFO:teuthology.orchestra.run.smithi067.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39001.log.gz 2024-03-29T11:30:33.184 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39625.log 2024-03-29T11:30:33.184 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34718.log.gz 2024-03-29T11:30:33.184 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40172.log: gzip 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41345.log.gz 2024-03-29T11:30:33.184 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40253.log 2024-03-29T11:30:33.184 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39625.log: gzip 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40172.log.gz 2024-03-29T11:30:33.184 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40409.log 2024-03-29T11:30:33.184 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40253.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39625.log.gz 2024-03-29T11:30:33.184 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39547.log 2024-03-29T11:30:33.184 INFO:teuthology.orchestra.run.smithi067.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40253.log.gz 2024-03-29T11:30:33.184 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40409.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:30:33.184 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40292.log 2024-03-29T11:30:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39547.log: /var/log/ceph/ceph-mgr.y.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40409.log.gz 2024-03-29T11:30:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40760.log 2024-03-29T11:30:33.185 INFO:teuthology.orchestra.run.smithi067.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39547.log.gz 2024-03-29T11:30:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38062.log 2024-03-29T11:30:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40760.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:30:33.185 INFO:teuthology.orchestra.run.smithi067.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40292.log.gz 2024-03-29T11:30:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34615.log 2024-03-29T11:30:33.185 INFO:teuthology.orchestra.run.smithi067.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40760.log.gz 2024-03-29T11:30:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38062.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38218.log 2024-03-29T11:30:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34615.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38062.log.gz 2024-03-29T11:30:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40721.log 2024-03-29T11:30:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34615.log.gz 2024-03-29T11:30:33.185 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38572.log 2024-03-29T11:30:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40721.log: /var/log/ceph/ceph-client.admin.38218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38767.log 2024-03-29T11:30:33.186 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38572.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40721.log.gz 2024-03-29T11:30:33.186 INFO:teuthology.orchestra.run.smithi067.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38218.log.gz 2024-03-29T11:30:33.186 INFO:teuthology.orchestra.run.smithi067.stderr:gzip 86.2% -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:30:33.186 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38572.log.gz 2024-03-29T11:30:33.186 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40604.log 2024-03-29T11:30:33.186 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38767.log.gz 2024-03-29T11:30:33.186 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40565.log 2024-03-29T11:30:33.186 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40604.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40604.log.gz 2024-03-29T11:30:33.186 INFO:teuthology.orchestra.run.smithi067.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:30:33.186 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40448.log 2024-03-29T11:30:33.186 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:30:33.186 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40565.log: gzip/var/log/ceph/ceph-client.admin.40448.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38455.log 2024-03-29T11:30:33.186 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.2.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40448.log.gz 2024-03-29T11:30:33.186 INFO:teuthology.orchestra.run.smithi067.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40565.log.gz 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41111.log 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39313.log 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38455.log.gz 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39781.log 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39313.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41111.log.gz 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39157.log 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39313.log.gz 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38884.log 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39157.log: /var/log/ceph/ceph-client.admin.39781.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39781.log.gz 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39157.log.gz 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40526.log 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39586.log 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38884.log: /var/log/ceph/ceph-client.admin.40526.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38884.log.gz 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40526.log.gz 2024-03-29T11:30:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39937.log 2024-03-29T11:30:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38650.log 2024-03-29T11:30:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39937.log: /var/log/ceph/ceph-client.admin.39586.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39586.log.gz 2024-03-29T11:30:33.188 INFO:teuthology.orchestra.run.smithi067.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39937.log.gz 2024-03-29T11:30:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40331.log 2024-03-29T11:30:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40132.log 2024-03-29T11:30:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40331.log: /var/log/ceph/ceph-client.admin.38650.log: 85.6% 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38650.log.gz 2024-03-29T11:30:33.188 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40331.log.gz 2024-03-29T11:30:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38179.log 2024-03-29T11:30:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38689.log 2024-03-29T11:30:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40132.log: /var/log/ceph/ceph-client.admin.38179.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40132.log.gz 2024-03-29T11:30:33.188 INFO:teuthology.orchestra.run.smithi067.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38179.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41150.log 2024-03-29T11:30:33.188 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:30:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39196.log 2024-03-29T11:30:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38689.log: /var/log/ceph/ceph-client.admin.41150.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41150.log.gz 2024-03-29T11:30:33.189 INFO:teuthology.orchestra.run.smithi067.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38689.log.gz 2024-03-29T11:30:33.189 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40054.log 2024-03-29T11:30:33.189 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38923.log 2024-03-29T11:30:33.189 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39196.log: /var/log/ceph/ceph-client.admin.40054.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39196.log.gz 2024-03-29T11:30:33.189 INFO:teuthology.orchestra.run.smithi067.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40054.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:30:33.189 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:30:33.189 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38377.log 2024-03-29T11:30:33.189 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38923.log: /var/log/ceph/ceph-mon.c.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38923.log.gz 2024-03-29T11:30:33.192 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38377.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38377.log.gz 2024-03-29T11:30:33.193 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:30:33.193 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34788.log 2024-03-29T11:30:33.193 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34868.log 2024-03-29T11:30:33.193 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.34788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34988.log 2024-03-29T11:30:33.193 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.34868.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:30:33.193 INFO:teuthology.orchestra.run.smithi082.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34788.log.gz 2024-03-29T11:30:33.193 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.34988.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:30:33.193 INFO:teuthology.orchestra.run.smithi082.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34868.log.gz 2024-03-29T11:30:33.193 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34828.log 2024-03-29T11:30:33.194 INFO:teuthology.orchestra.run.smithi082.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34988.log.gz 2024-03-29T11:30:33.194 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:30:33.194 INFO:teuthology.orchestra.run.smithi082.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:30:33.194 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.34828.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:30:33.194 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34748.log 2024-03-29T11:30:33.194 INFO:teuthology.orchestra.run.smithi082.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34828.log.gz 2024-03-29T11:30:33.194 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35028.log 2024-03-29T11:30:33.194 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.34748.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:30:33.194 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.35028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34948.log 2024-03-29T11:30:33.194 INFO:teuthology.orchestra.run.smithi082.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34748.log.gz 2024-03-29T11:30:33.194 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:30:33.194 INFO:teuthology.orchestra.run.smithi082.stderr: 87.9%/var/log/ceph/ceph-mgr.x.log: -- replaced with /var/log/ceph/ceph-client.admin.35028.log.gz 2024-03-29T11:30:33.194 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.34948.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:30:33.195 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34948.log.gz 2024-03-29T11:30:33.195 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34908.log 2024-03-29T11:30:33.195 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-mon.b.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:30:33.195 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.34908.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34908.log.gz 2024-03-29T11:30:33.195 INFO:teuthology.orchestra.run.smithi082.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:30:33.205 INFO:teuthology.orchestra.run.smithi067.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:30:45.245 INFO:teuthology.orchestra.run.smithi082.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:30:45.275 INFO:teuthology.orchestra.run.smithi082.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:30:47.922 INFO:teuthology.orchestra.run.smithi067.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:30:49.265 INFO:teuthology.orchestra.run.smithi082.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:30:49.443 INFO:teuthology.orchestra.run.smithi082.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:30:50.249 INFO:teuthology.orchestra.run.smithi067.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:30:51.248 INFO:teuthology.orchestra.run.smithi067.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:30:51.257 INFO:teuthology.orchestra.run.smithi067.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:31:24.213 INFO:teuthology.orchestra.run.smithi067.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:31:27.043 INFO:teuthology.orchestra.run.smithi067.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:31:27.044 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-29T11:31:27.045 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m53.926s 2024-03-29T11:31:27.045 INFO:teuthology.orchestra.run.smithi067.stderr:user 2m47.427s 2024-03-29T11:31:27.045 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m5.422s 2024-03-29T11:31:40.246 INFO:teuthology.orchestra.run.smithi082.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:31:40.248 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:31:40.248 INFO:teuthology.orchestra.run.smithi082.stderr:real 1m7.090s 2024-03-29T11:31:40.248 INFO:teuthology.orchestra.run.smithi082.stderr:user 1m59.079s 2024-03-29T11:31:40.248 INFO:teuthology.orchestra.run.smithi082.stderr:sys 0m4.347s 2024-03-29T11:31:40.248 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:31:40.249 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693/remote/smithi067/log 2024-03-29T11:31:40.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:07.549 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693/remote/smithi082/log 2024-03-29T11:32:07.551 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:26.962 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:27.175 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1105cde148554088ab3f7d159cc712fc Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:27.182 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:32:27.194 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:32:27.215 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:32:27.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:27.230 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:27.309 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:32:27.327 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:32:27.327 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:27.330 DEBUG:teuthology.orchestra.run.smithi082:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:27.344 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:27.346 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:27.349 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:27.349 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-03-29T11:32:27.349 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 27 +464us[ +464us] +/- 92ms 2024-03-29T11:32:27.349 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -105us[ -111us] +/- 54ms 2024-03-29T11:32:27.349 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -258us[ -258us] +/- 86ms 2024-03-29T11:32:27.349 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:27.350 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:27.351 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-03-29T11:32:27.351 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 +461us[ +454us] +/- 92ms 2024-03-29T11:32:27.351 INFO:teuthology.orchestra.run.smithi082.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -113us[ -120us] +/- 54ms 2024-03-29T11:32:27.351 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 -248us[ -255us] +/- 86ms 2024-03-29T11:32:27.351 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:27.352 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:32:27.369 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:32:27.369 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:32:27.386 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:27.460 DEBUG:teuthology.orchestra.run.smithi082:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:27.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:27.560 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:27.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:27.592 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:32:27.601 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:32:27.615 INFO:teuthology.task.internal:Duration was 1275.808330 seconds 2024-03-29T11:32:27.615 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:32:27.629 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:32:27.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:27.632 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:27.674 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:27.675 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:28.068 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:32:28.069 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-03-29T11:32:28.069 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:28.090 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T11:32:28.090 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:28.134 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:32:28.135 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:28.137 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:28.203 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:32:28.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:28.206 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:28.340 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:32:28.350 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:32:28.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:28.378 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:28.407 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:32:28.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:28.422 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:28.445 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-03-29T11:32:28.472 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2024-03-29T11:32:28.497 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:28.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:28.521 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:28.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:28.552 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:32:28.563 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:32:28.564 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693/remote/smithi067 2024-03-29T11:32:28.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:28.600 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693/remote/smithi082 2024-03-29T11:32:28.601 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:28.636 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:32:28.636 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:28.638 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:28.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:32:28.695 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:32:28.696 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:32:28.706 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:32:28.707 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:28.709 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:28.721 INFO:teuthology.orchestra.run.smithi067.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:28.739 INFO:teuthology.orchestra.run.smithi082.stdout: 266042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:28.740 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:32:28.765 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:32:28.906 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:32:28.939 DEBUG:teuthology.nuke:shortname: smithi067 2024-03-29T11:32:28.939 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:28.966 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:08.879224', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOWEyAUDB6nKarlTJ77qio03BlccevtgBeAzfG88npriw1GXRuGLuaygY7KdT1Cok+NjmfiQsE7OoEF4dEyVwKc='} 2024-03-29T11:32:28.967 DEBUG:teuthology.nuke:shortname: smithi082 2024-03-29T11:32:28.968 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:28.996 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622693', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:08.877644', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPnLy+p2JFVioUa8/WLiHzMs21gwvt5OFzRAMN/H+2A0+xE4eK/hll2PJ4DvI6L8aWJb9ENYFvMXZgSywlqtsw0='} 2024-03-29T11:32:29.026 INFO:teuthology.orchestra.console.smithi067:Power off 2024-03-29T11:32:29.026 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:29.068 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:29.068 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:29.090 INFO:teuthology.orchestra.console.smithi082:Power off 2024-03-29T11:32:29.090 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:29.106 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-03-29T11:32:29.128 DEBUG:teuthology.orchestra.console.smithi082:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:29.128 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:29.161 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-03-29T11:32:33.108 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:33.705 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:33.807 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-03-29T11:32:33.818 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-03-29T11:32:37.811 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:37.821 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:37.923 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2024-03-29T11:32:37.934 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off 2024-03-29T11:32:38.024 INFO:teuthology.orchestra.console.smithi067:Power off completed 2024-03-29T11:32:38.035 INFO:teuthology.orchestra.console.smithi082:Power off completed 2024-03-29T11:32:38.140 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1275.80833029747 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1105cde148554088ab3f7d159cc712fc status: fail success: false 2024-03-29T11:32:38.140 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:32:38.223 INFO:teuthology.run:FAIL